Advancing Expert Specialization for Better MoE
Authors: Hongcan Guo
Haolang Lu
Guoshun Nan
Bolun Chu
Jialin Zhuang
Yuan Yang
Wenhao Che
Xinye Cao
Sicong Leng
Qimei Cui
Xudong Jiang
Venue: NeurIPS 2025 oral ()
Keywords: Mixture of Experts
Load Balancing
Optimization
Supervised Fine-Tuning
Primary area: deep_learning
TL;DR: Our proposed orthogonality and variance losses improve performance in downstream fine-tuning of Mixture-of-Experts models by enhancing expert specificity, addressing expert homogenization caused by load balancing, while maintaining load balance.

Abstract: Mixture-of-Experts (MoE) models enable efficient scaling of large language models (LLMs) by activating only a subset of experts per input. 
However, we observe that the commonly used auxiliary load balancing loss often leads to expert overlap and overly uniform routing, which hinders expert specialization and degrades overall performance during post-training.
To address this, we propose a simple yet effective solution that introduces two complementary objectives: (1) an orthogonality loss to encourage experts to process distinct types of tokens, and (2) a variance loss to encourage more discriminative routing decisions.
Gradient-level analysis demonstrates that these objectives are compatible with the existing auxiliary loss and contribute to optimizing the training process.
Experimental results over various model architectures and across multiple benchmarks show that our method significantly enhances expert specialization. 
Notably, our method improves classic MoE baselines with auxiliary loss by up to 23.79\%, while also maintaining load balancing in downstream tasks, without any architectural modifications or additional components. We will release our code to contribute to the community.

Advancing Expert Specialization for Better MoE
Hongcan Guo
Haolang Lu
Guoshun Nan It
Bolun Chu
Jialin Zhuang'
Yuan Yang'
Wenhao Che
Xinye Cao'
Sicong Leng"
Qimei Cui
Xudong Jiang?
BBejjing University of Posts and Telecommunications, China
Nanyang Technological University, Singapore
{ai {ai.guohc, guohc, 1h1__501000 manguo2021)@bupp edu. cn
Abstract
Mixture-of-Expert (MoE) models enable efficient scaling of large language mod- els (LLMs) by activating only a subset of experts per input. However, we observe that the commonly used auxiliary load balancing loss often leads to expert over- lap and overly uniform routing, which hinders expert specialization and degrades overall performance during post-training. To address this, propose simple yet effective solution that introduces two complementary objectives: (1) an or- thogonality loss encourage experts to process distinct types of tokens, and (2) variance loss to encourage more discriminative routing decisions. Gradient-level analysis demonstrates that these objectives compatible with the existing auxil- iary loss and contribute to optimizing the training process. Experimental results over various model architectures and across multiple benchmarks show that our method significantly enhances expert specialization. Notably, our method improves classic MoE baselines with auxiliary loss by up 23.79%, while also maintaining load balancing in downstream tasks, without any architectural modifications or additional components. Our code is available at this link.
Introduction
Large language models (LLMs) [67, 65, 62, 6] have demonstrated remarkable generalization capa- bilities [52, 69, 74, 73] across a wide range of tasks [53, 24], but their inference cost [15, 57] grows rapidly with scale, hindering practical deployment and efficiency. Mixture-of- of--eeettstteeeeeoooo (MoE) [9, 3, 37] architectures alleviate this problem by activating only a subset of experts per input [19], thus enabling
greater model capacity without commensurate increase in computational overhead [22, 49, 33]. To maximize parameter utilization, MoE systems typically introduce load balancing [56, 20] objectives that encourage more uniform routing of tokens across experts during pre-training.
While load balancing effective in avoiding idle experts during large-scale pre-training, it often hinders model adaptation in the post-training stage for downstream tasks, where data distributions are narrower and more fomain-specifee In such settings, token occurrences are typically concentrated within particular subspaces (e.g., numeric symbolic tokens in math tasks), intensifying the tension between balanced routing and expert specialization. A widely observed phenomenon is that load bulancing encourages uniform expert routing across inputs, resulting in highly overlapping token distributions [14, 79]. This overlap leads to convergence in expert representations [46], ultimately compromising the development of specialized functionalities. The lack of specialization [14] becomes particularly problematic during fine-tuning [17, 60, 2, 80] on downstream tasks with strong domain preferences, where the model struggles to adapt and exhibits degraded performance [34].
'Equal contribution. CCorresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ronting Diverafication
Expert Specialiaatinn,..
<- - Reney
Radg (Aa Roy
Ronting Store Metrle
Experts Tisken
Balance
Aeguon Mive
Caod Erarhane Deernee
Reuting Ouiput Diverse
coulteuting Output
Expera Balee Vrea
Diverse (Trainins)
OVERLAP
Two core effects of our method. Left - Routing Diversification: Left-Boode Bottom: after training, scores show higher discrimination than the untrained model. Right-Top: expert load variance decrease after training. Right-Boot Bottom: when training, variance increases markedly, yielding more decisive token-to-expert assignments. Right Expert Specialization: Cluster Separation: clearer per-expert token clusters emerge after training, evidencing specialization. Overlap: baseline exhibits heavy token-assignment overlap across experts, which method substantially reduces.
Figure I:
This highlights a core challenge in MoE post-training: the inherent conflict between encouraging expert specialization [50, 38, 36]and enforcing routing uniformity [83] via auxiliary losses. From the expert perspective, brad-balanced routing causes overlapping training intentions across experts [14, 45, 46, 7], suppressing the development of distinct expert behaviors. From the router perspective, as experts become less specialized, the router receives less variation across experts, leading to increasingly uniform and less informed token-to-expee to-eeertetpttteeee assignments [82]. These dynamics form self-reinforcing loop: diminished specialization and uniform routing exacerbate each other over time, progressively degrading both expert expressiveness and routing quality [20]. This compounding effect reveals a deeper limitation of existing training objectives, which lack mechanisms to decouple expert specialization from the uniformity constraints imposed by auxiliary losses.
To address this challenge, we propose gradient-bact based multi-objective optimization framework that promotes expert specialization and routing diversification, while preserving load balance from auxiliary loss. We introduce two complementary objectives, as shown in Figure 1: 1) Expert Specialization, which fosters distinct expert representations by ensuring that each expert specializes in processing different tokens. 2) Routing Diversification, which drives differentiated routing decisions, enabling more precise token-to-expert assignments by enhancing the variance in routing. By jointly optimizing these objectives, our method mitigates the trade-off between model performance and routing efficiency MoE. training. We demonstrate that our approach successfully achieves:
Enhanced expert-routing synergy. Our joint objectives reduce expert overlap by up to 45% and increase routing score variance by over 150%, leading clearer specialization and more discriminative expert assignment.
Stable load balancing. Despite introducing new objectives, our method matches the baseline's Max Vioglobal across all models, with RMSE under 8.63 in each case.
Improved downstream performance. We achieve 23.79% relative gains across 11 benchmarks and outperform all baselines on 92.42% of tasks ,all without modifying the MoE architecture.
Motivation
2.1 Preliminaries of MoE
In typical MoE layer, let there be experts, and sequence of input tokens represented by X= (I1, I2, ], where N is the total number of tokens the sequence. The routing score matrix after applying the top-k mechanism is denoted as:
$11
$12
S1n
$21
822
82n
1 Siz
S
1, i i= 1,1,2 1.2. ..,
(1)
8N1
SN2
S

---

where Sij represents the routing weight assigned to the i-th token for the j-th expert.
Let F == (J1, J2..... Jn) represent the proportion of tokens assigned each expert, where fj is the number of tokens assigned to the j-th expert. For any given MoF. layer, the total loss function L. consists of two parts, the main loss La. and the auxiliary loss Louz:
N
L == La + a Laux = Ln + Σ f3 Ps. ==1
8ij.
(2)
where LA is the loss computed from the output of the MoE layer. and Lauz is the auxiliary loss term, denotes the weighting coefficient for the auxiliary loss. Here, Pj represents the total routing score for the j-th expert, which is the sum of the routing weights for all tokens assigned to that expert.
2.2 Observations
Obs I (Expert Overlap): Introduction of the auxiliary loss function leads to a more homogenized distribution of tokens across experts, which may reduce the distinctiveness of each expert.
It has been observed that the auxiliary loss function is independent of the expert parameter matrices g Therefore, for the j-th expert, its gradient can be written as:
ac aCn + 80F, 80 30p₂
dLaux Oc Tyh 00F;
N
Σ
24 S.j.J
1,2, ,
n2.
Oyh
where DF, is the parameter matrix of the j-th expert, and yı. is the output of the MoE layer. During gradient descent, the addition of the auxiliary loss Lour forces the routing mechanism to evenly distribute the tokens across experts as much as possible.
This results in input token z; being assigned to an expert that may not be semantically aligned with it, causing unintended gradient flow to expert j. Mathematically, after applying the top-k mechanism, the routing score 8ú transitions from 0 to a non-zero value, introducing gradients from tokens that originally had no affinity with expert j-
Obs II (Routing Uniformity): As training progresses, the routing output tends to become more uniform, with the expert weight distribution gradually converging towards equal allocation.
To understand this phenomenon, we first examine the source of gradients with respect the routing parameters fg. Since the routing mechanism produces only the score matrix S = Sij, the gradient 0c/00R can be written as:
aL dhh
DLouz 00R
086 R
J8k 80
Σ
f3
00R
D0R
where r; - FF represents the output of expert j for token v,, and f, denotes the frequency with which expert j selected. This formulation reveals that the routing gradient primarily influenced by the expert outputs and the token distribution across experts.
The auxiliary loss Laux introduced to encourage balanced token assignment by optimizing the uniformity of fy. However, since f, is non-differentiable, direct optimization is not feasible. Instead, surrogate variable Pj. which differentiable and positively correlated with fz. is employed to approximate the objective and enable gradient flow back to the routing network.
As training proceeds, the optimization objective increasingly favors the uniformity of pý. which drives fy toward an even distribution. Moreover, as discussed in Observation I, incorrect token assignments caused by auxiliary regularization introduce overlapping gradients among experts, increasing the similarity of zę OF, across different j.
Obs III Exppert-Routing Interaction): While Obs I concerns expert specialization, while Obs II
reflects the uniformity of routing. These two effects interact during training, jointly driving the model toward degraded performance.
Expert-side interference caused by Obs leads to blurred specialization. Tokens are assigned to mismatched experts, and the resulting gradient interference reduces expert distinctiveness. As the
3

---

routing weights become more uniform, different experts receive similar gradients from the same tokens, increasing their functional overlap.
This expert similarity feeds back into the routing mechanism. As expert outputs become less distinguishable, the routing network finds fewer cues to differentiate among experts, leading even more uniform weight distributions. This promotes random top-k selection and further misalignment between tokens and their optimal experts.
Together, this loop gradually steers the model toward more uniform token allocation and reduced expert specialization, highlighting potential opportunities for improving the routing strategy and expert assignment.
3 Method
Based on the observations above, we propose the following design to mitigate expert overlap and routing uniformity, the overall loss function C is defined as follows:
C == Ln + Lialance, Lbalance = a Lauz + B. - 7- Ly:
(S)
where Laux represents the existing auxiliary loss, with coefficient a, and the newly introduced orthogonality loss C, and variance loss L. (see Subsec 3.1), with coefficients 3 and yy respectively. is worth noting that the theoretical complementarity of these optimization objectives, rather than any inherent conflict, is formally analyzed and demonstrated in Subsection 3.2.
3.1 Implementations of Losses Co and Lu
In this section, introduce two critical loss functions Lo and Lu that act on the expert and router components, respectively.
Expert Specialization. We introduce an orthogonalization objective that encourages independent expert representations. Specifically, we design the following orthogonality loss:
N
(i Tak)) (aa) +
C.
ži = Ag I[ay>0)>
where (-) denotes the inner product between two vectors, and Isij > 0 is an indicator function that evaluates to 1 when Sty > 0 and 0 otherwise. Here, ē.y represents the output of expert for token I; after the top-k routing selection.
The orthogonality loss L. reduces the overlap between different expert outputs within the same top-k group by minimizing their projections onto each other. This encourages experts to develop more distinct representations, promoting specialization in processing different token types.
Routing Diversification. We introduce variance-based loss to encourage more diverse routing decisions and promote expert specialization. Specifically, we define the variance loss as:
C.
(sig 8,)
8Kjs
N
where 8j denotes the average routing score for expert across the batch. By maximizing the variance of routing scores, L. discourages uniform token-to-expert assignments and encourages more deterministic and distinct routing patterns, thereby facilitating expert specialization.
3.2 Compatibility of Multi-Objective Optimization
In this section, we analyze how each component influences the optimization dynamics of expert parameters 8E, and routing parameters Hµ during training. Meanwhile, will focus the optimization and compatibility the two losses L, and L, with respect to load balancing and expert specificity. The following two key questions guide our analysis.
Balancing Expert and Routing. How can expert (Lo) and routing (Lo) optimizations designed to complement each other without compromising their respective objectives?

---

We first demonstrate that L. and L are compatible in their optimization directions within MoE, then show that they mutually reinforce each other.
Mutually Compatible. We elaborate on the compatibility of La and Lu from the perspectives of expert and Routing.
From the expert perspective, we observe that the auxiliary loss Lauz and the variance loss C, do not directly contribute gradients to the expert parameter matrix E Consequently, the gradient of the total loss with respect to gg, is derived solely from the primary task loss Cn, and the orthogonality loss Co:
Oc F
T (iake Take) +
Sij Iv: +B.
Dij
Here, 9y, v Cn denotes the gradient the primary task loss with respect to the model output. This gradient is influenced by both the routing score sij and the expert representation ē.j. As training progress, the variance expert weights increases, and the gradient encourages stronger preferences in different directions for each token.
From the routing perspective, we notice that C. does not affect the gradient with respect to routing parameters 2g. The gradient of the total loss with respect to f is:
OC =
0C 08:j
N
aa. - 2(N 1) (8.j (8.)-85) -))) nN
0sg 00R
ΣΣ i=1j=1
00R Os.j
00R
=
(9)
This gradient is influenced by expert representations žij. expert load fj. and routing weights stj. As the model converges, the expert load f becomes more balanced, and the variance of routing weights Si increases. Orthogonalizing expert representations causes the routing gradients to flow more orthogonal directions, making the weight allocation more biased towards the representations and increasing the weight variance.
Summary. Expert parameters i,, are solely influenced by the gradients of C. without conflict. While routing parameters On are affected by both L. and L,, the objectives of these two losses (orthogonality- friendliness vs. score diversification) remain non-conffictngg
Mutually Reinforcing, C. aims to encourage the effective output vectors of different selected experts j and ls to tend be orthogonal for the same input token F, i.e., (ū.j Rss)) Rs The learning signal for the routing mechanism partially originates from the gradient of the primary task loss Lb. with respect the routing score Siz:
ac
DLax
2(N - 1) n.V
aCr Oy:
Osij
T Tg.
Tig +
dsij
8z);
U1 Σ Stj
9v:
(10)
from Ca.
from
Ear
from C.
Assuming Pij Sy. Zij. when the expert outputs tend to be orthogonal, for any given task gradient Py.- the projections Pij onto these approximately orthogonal expert outputs are more likely to exhibit significant differences. The increased variance of the primary relateel signals Piy implies that the routing mechanism receives more discriminative and stronger learning signals, which creates more favorable conditions for L. to achieve diversification of routing scores.
f.p enhances the diversity of routing scores Sij by optimizing routing parameters Oz. Meanwhile, due to the influence of L. gradient 8 8&6. on 0R, routing tends to assign more specialized token subsets Tj to each expert j. Expert parameters dE, learn the unique features of tokens within T₁, leading to gradual functional divergence among experts, thereby promoting expert orthogonality.
Summary, L. induces orthogonal expert outputs æay, enhances the discriminative power of routing signals 9y, F.. and generates diverse routing scores s;j to support L.. Meanwhile, L, drives experts to specialize in distinct token subsets via sij and promotes parameter divergence of 0Rg support L... Together, they form mutually reinforcing cycle.
T
5

---

Multi- Optimization. How do expert and routing maintain their balance while enhancing Lauz and Ln independently, ensuring nutually beneficial performance improvements?
Lemma Let S E R. x be a matrix that satisfies following conditions: each row sums to each row contains k non-zero elements and n. n-- k zero elements. Then, there always exists state in which the following two objectives are simultaneously optimized: The sum of the elements each column tends to the average value N. 2. The variance of the non-zero elements in each row increases.
Lemma For two sets of points A and B of equal size, it is always possible to partition A U B such that An B = Ø and [A] [B].
The overall objective function L optimizes four key dimensions: accurate data fitting(Cs), expert orthogonalization((a.) balanced expert routing weights(Laaur,, and increased variance in routing outputs(C.)) Our core objective is to achieve an optimal balance by jointly optimizing these multiple objectives, ensuring they complement each other for enhanced model performance.
As shown by Lemma 1, expert load fj and routing weights Sıj can be optimized together. As demonstrated in Lemma 2, the objectives of orthogonalization and load balancing are not in conflict and can jointly optimized. Thus, both expert and routing modifications can be optimized alongside load balancing (balanced expert routing weights).
Moreover, orthogonalization enhances routing weight variance, in turn, improves expert specialization (as discussed in Section 2.2). This leads to more distinctive expert representations, aligning with performance (accurate data fitting) improvements when optimized together.
4 Experiments
In this section, we conduct experiments to address the following research questions:
Does introducing the orthogonality loss (Co) and variance loss (C.) lead to better overall performance downstream tasks compared to baseline approaches?
RQ1:
RQ2: To what extent does our method maintain expert load balancing during training?
RQ3: How do the orthogonality loss (Co) and variance loss (C,) interact with each other, and what are their respective and joint impacts on expert specialization and routing behavior?
RQ4: What are the individual and combined contributions of Co, Cv. and the auxiliary loss Loux the final model performance?
4.1 Experimental Setup
Environment. All experiments are performed on CentOS Linux 7 server with PyToreh 2.3. The hardware specifications consist of 240GB of RAM, a 16-core Intel Xeon CPU, and two NVIDIA A800 GPUs, each having 80GB of memory. Implementation details are provided in the Appendix F.
Datasets. We evaluate method total of 11 benchmarks. Specifically,
the training sets from Numina [41], GLUE [66], and the FLAN collection [72] to train our models. Our bench-
marks include: O Mathematics: GSM8K [12], MATH500 [44], and Numina [41]: @ Multi-Domain Tasks: MMLU [31, 30), MMLU-pro [70], BBH [63], GLUE [66]; LiveBench [76] and GPQA [59].
@ Code generation: HumanEval [10] and MBPP [4]. We group training and test sets by language, reasoning, science, math, and code to match downstream evaluation needs. Detail in Appendix D.
Baselines. We compare our method with 4 existing MoE training strategies. With Aux Loss [46] applies auxiliary load-balancing losses during routing to encourage expert utilization diversity. GShard [39] introduces foundational sparse expert framework with automatic sharding and rout- ing; ST-MoE [85] enhances training stability via router dropout and auxiliary losses; Loee-Free Balancing [68] achieves balanced expert routing without auxiliary objectives. Detail in Appendix G.
Metrics. We employ 6 evaluation metrics to test our method in terms of accuracy, expert load balaneing (Max ViOgiobal [68]), clustering quality (Silhouette Coefficient), expert specialization (Expert MMxXViogtaa
Overlap), routing stability (Routing Variance), and prediction error (RMSE). Detail in Appendix E.

---

Table 1: Performance on different downstream tasks. The table shows accuracies of methods across models and downstream tasks. Notably, we categorize sub-downstream tasks in Multi-Domain and ensure training/evaluatinn sets are domain-aligned, following downstream task requirements.
Multi Dollii-Duuni (Arg.)
Code
Mach
Mathod
Modil
MMLU
MMLLEppr
BBH
GLUE
Lävehench
GPOA
HumunEval
MBP
GSMHK
MATH550
Numina
With Aux Loss
29.27
19.47zo
26.92zz
49 a
7,83gm
25
51.52a
31.36:
15.70
5,47a1
14.99.z
Looo-Poor Balancing
Deepleck-
30,71
16.8
32.99pm
49.60:1
9,70pmm
20.63
53.16
32,80:-
21.28
5.83
17:25
GShurd
27:05
20.45...
23
3.00
24.28
57.75
34,50
27.12
8.20
16.99
ST-MO
34,23
m9.7
36.91.3
54. 56
6.88pm
20 35.3
53.28
36.34:
30 10
7.08
15
Ours
33.35
24,87
60.11
1m
25.
63.30
40.83
35.00
10.52
20.41 am
Loss
19:23
35.07um
11,70
42.00
Leoa-Fice Belaninn
DecpBook
30.23
30.75.3.4
34.2
39.83...
10.15
26 4..3
41.28:
36.02.o
43.35
50.76 30.76
43 a3
GShard
10 21-66:
97.07
16.80.1
48.36.1
43.61
ST-MOE
32 :0.68....
30.28:
38.78
38.27
10
22. 33
46.10oam
39.72
47.78
46, 74
48 65.55
37.37
3884
41.20
43.58
43.53
58/94,1a
49.M
50.67
With Aus Lows
35 ar
36.10.1w
47. 47.17
26.16
15.84
30.72r
63.61:w
47.34:
57
45.4
Lows-Peee Batancing
Moonlgght
150 13
27 w
42.45
32.97.M
20,06a2
29.27 29.27
62.93u2
44.22
79,54
59.77
42 42.8
GShard
16 16.06uss
10.65aw
49.20e17
34. 34.46o2a
13.92
31. 13rre
64.50
49.85.am
84 84.02
56
47 18
ST-MOE
39.03
26.85.
48.78
30.18:1
16.99
30
06.04.0
4797
84.45
57,51
49 42:2
Ours
40.36
34.90.a
52.42.1a
37.01:314
20.88
32.01:xm
70.64:4w
47.77=
87.62
59,64
52.8k
Setup. Each benchmark is fine-tuned separately on 6,000 high-quality examples, primarily from the official training split and supplemented when necessary. Answers are generated using strong teacher models (OpenAl o3-mini and DeepSeek R1) and manually verified for correctness. Fine-tuning is limited to three epochs (~550 steps) to prevent overfitting.
All experiments adopt LoRA-based fine-tuning, with LoRA modules inserted into both router and expert layers to enable joint optimization. A rank of 32 is used to approximate full-model updates. Detailed configurations, including optimizer, batch size, and learning rate, are provided in Appendix H.2.
4.2 Performance in Downstream Tasks (RQ1)
To verify that our Lislance enhances model performance in downstream task scenarios through expert orthogonality and routing output diversification, as shown in Table 1, we design downstream task scenarios on 11 well-known benchmarks and validate our method against four baseline methods with distinct loss designs on three widely used MoE models. We make the following observations:
Obs. Baseline methods without guidance for expert specialization exhibit varied performance and fail to effectively improve downstream task performance. As shown in Table 1, the four baseline methods show no clear overall performance ranking across the 11 tasks, with performance variations within 2% in many tasks. Their overall performance is significantly lower than our method, demonstrating no potential to improve downstream task performance.
Obs. e Our method guiding expert specialization effectively enhances model performance in downstream tasks. As shown in Table I, we achieve state-of-the-ar art (SOTA) results in over 85% of the 33 tasks across the three models. In some tasks, the average across multiple measurements outperforms the next-best method by nearly 7%. Extensive experiments indicate that our method significantly improves model performance in downstream task scenarios by enhancing expert specialization. More results additional baselines and MoE architectures provided in Appendix I.
4.3
Load Balancing (RQ2)
To verify that our newly added losses E. and Co do not affect the load balancing effect, we conduct statistical measurements on the load balancing of all combinations of Laue, Cy, and C, across various models during training.
Figure shows the variation of MarViogttta I across training steps for different loss combinations, as well as the RMSE of differences between our method and other combinations. We make the following observations:
Obs. H Loss combinations without Laux exhibit significantly worse load balancing performance than those with Laux- As shown in Figure 2, across three distinct models, the M Marl V iO of the w/o all method (with no losses added) is significantly higher than that of other methods, indicating
7

---

DeepSeck-Moo Moe-loB
Moonlight-16B 1B----
DeepSeck-V2 Vi-Lite
Mortend
shhttend
Stattend
S143
Step
Step
Figure 2: Variation of Load Balancing. The figure illustrates the variation of load balancing during training across three distinct models for different methods. Method represents the combination of Laux. Lo. and L.; Step denotes the number of training steps; MaxV 'Ogioball serves as the metric for load balancing; and RMSE is the metric for measuring the similarity between two curves.
notably poorer load balancing. In particular, for the DeepSeek-v2-Lite model, the method without Laux converges to 6.14, whereas methods with Laux converge to 2.48. demonstrating that loss combinations containing Laux achieve significantly better load balancing.
Obs. Incorporating any combination of L. and L., into Laux does not affect load balancing. As shown in Figure 2, for methods with Laxx: the trends of ""nlyy aux" (no additional losses), "w/o Iv" (only L.). "w/o lo" (only Ly), and "ours" (both L. and Lo) nearly identical. Additionally, the RMSE (root mean squared error) of our method relative to other baselines does not exceed 0.03, further corroborating the conclusion that the combination of C,, and L. does not impact load balancing.
4.4 Behaviors of Experts and Routing (RQ3)
To verify that L, and Lo can jointly promote expert orthogonality and routing diversification, following the method setup in Section 4.3, we will conduct evaluations expert orthogonality and measurements of routing score diversification for different loss combinations.
Silsouete Confficieut
Expert Ovedl Compartonn
Ronting Varianee Comparison
voal
Modat
Mode
Madel
1003 000-04
silbourtte coeffictent
Figure 3: Behaviors of Experts and Routing. The figure demonstrates the behavioral states of experts and routing across different methods. The first two subplots, Silhouette Coefficient and Expert Overlap, measure the degree of expert orthogonality, while the last subplor, Routing Variance, evaluates the diversity of routing outputs.
As shown in Figure 3, the first two subplots demonstrate the orthogonality of experts, while the last subplot illustrates the diversification of routing outputs. We make the following observations:
Obs. E. directly promotes expert orthogonality, and C. also aids in expert orthogonality. As shown in the first two panels of Figure 3, our method with both L. and C. achieves state-of-the t-thh-rat (SOTA) results across three models, with Expert Overlap even dropping below 0.3. The method with only C. and Laux (w/o Iv) consistently ranks second-best, indicating that L., has a more significant impact on expert orthogonality. Notably, the method with only L, and Lax (w/o lo) significantly outperforms the method with only Laxx across all three models, confirming that C,, also contributes to expert orthogonality.

---

Obs. L. directly enhances routing output diversification, and L. also supports this diversifi- cation. Similarly. our method exhibits the highest routing score variance (exceeding 0.010), followed by the method with only Lap and L while the method with only Laux performs worst. This strongly supports the conclusion.
Obs.0 Laux leads to higher expert overlap and more homogeneous routing outputs. Compared the w/o all method (no losses), the aux only method (with only Laux) shows Silhouette Coefficient that is over 0.05 higher and routing output variance that is 0.0045 higher. This indicates that w/o all exhibits significantly greater expert orthogonality and routing output diversification than aux only.
4.5 Ablation among Losses (RQ4)
To demonstrate that both L. and L. have positive effects on the model's performance in downstream task scenarios, and their combination synergistically enhances each other's efficacy, we design ablation experiments for these two losses on three models.
Ablation Experiment on Dec--eek-MMe-166
Ahlation Experiment on VeepSeek-V2-Lite
Ablation Experiment on Moonlight-168--N 6B-A3B
MMELU
Namitafle
MME.I:
NaminaTee
MMLII
NarinaTee
TRESO
TH503
65
50
100
.2
46
X00
CXNKK
I0II
G
BBI
0SXXX
G2.15
NBPP
GLA
NBPP
GLU
NBPP
GPQA
GPGA
GPGA
w/b
only
Figure 4: Ablation Experiments. The figure illustrates the performance differences of different ablation method combinations across three models on various benchmarks. The vertices on the circles represent the corresponding benchmark names, with the same type connected by the same color. The numbers inside the circles denote the accuracy represented by each circle.
Figure 4 illustrates the performance of different ablation method combinations across various down- stream tasks. We make the following observations:
O The combination of C, and C. significantly enhances model performance in downstream tasks, and each loss individually also improves performance. Our method (combining C, and C.) exhibits the largest coverage area across all three models, nearly encompassing other methods. When either L. or L, is ablated (i.e., w/o Iv or w/o lo), the coverage areas of these methods are larger than that of the only aux method (with only L indicating performance improvements over the baseline.
Obs. 2 Laux impacts model performance on downstream tasks. Figure 4 clearly shows that the only aux method (with only Laux) is nearly entirely enclosed by other methods across all three models, consistently exhibiting the smallest coverage area. Notably, the w/o all method (with no losses) achieves performance improvements and a larger coverage area than the only aux method when Caux removed, supporting this conclusion.
Beyond the ablation results in Fig. 4, we further conduct a sensitivity analysis on the loss-weight coefficients a, B, and y The detailed results and discussions are provided in Appendix H.1.
Related Work
Auxiliary Losses in MoE Training. Auxiliary losses [39, 85] are commonly used to prevent expert collapse by encouraging balanced expert utilization [14]. Early approaches focus on suppressing routing imbalance, while later works [81] introduce capacity constraints or multi-level objectives to separate routing stability from load balancing [65, 39, 20]. Recent methods [75] further reduce

---

manual tuning by dynamically adjusting auxiliary weights or replacing them with untropy-based routing [42]. However, fixed-rule strategies may underutilize expert capacity, and dynamic schemes can introduce instability or overhead, making robust balancing still a challenge [32, 68).
Orthogonality in MoE. Orthogonalization [47, 28] improves expert diversity by encouraging inde- pendent representations [29]. Some methods [54, 84, 51] regularize expert weights directly, while others [14, 29] assign experts to disentangled subspaces based on task semantics. Recent routing- based approaches [47, 58] also impose orthogonality on token-to-expert assignments to reduce redundancy. Nonetheless, static constraints [11] often fail to adapt to dynamic inputs, and dynamic ones [78, 35, 25, 64] may conflict with balancing, complicating expert allocation [32. 82, 27, 68]. Our work addresses these tensions by integrating orthogonalization and balance into a unified, gradient-consistent optimization framework.
6 Limitation & Future Discussion
While Lbatonce balances load and enhances performance in downstream tasks, its potential in other domains remains unexplored. Specifically, it could be extended to visual models, as suggested in recent work [26], and multimodal or full-modal settings [8] offering opportunities for cross- domain applications. Additionally, investigating Liatance within lightweight MoE fine-tuning, such LoRA-MoE [21], could make our approach viable for resource- cesource-constrained environments [43].
Furthermore, there is considerable potential in exploring expert-distributed deployment, where Lbelance can optimize both parameter inference efficiency and model performance. This avenue could significantly enhance the scalability and practicality of MoE models real-world applications, providing opportunities for distributed expert architectures.
Conclusion
In this work, present theoretically grounded framework that resolves the inherent conflict between expert specialization and routing uniformity in MoE training. By introducing orthogonality and variance-based objectives, our method significantly improves downstream performance without any architectural changes. This demonstrates that MoE efficiency and specialization can be simultaneously optimized through loss-level innovations alone. Experiments show the effectiveness of our method.
8 Acknowledgements
This work was supported in part by the National Key Research and Development Program China under Grant 2022YFB2902200; in part by the Guangxi Key Research and Development Program under Grant FN2504240005; in part by the National Natural Science Foundation of China under Grant 6247 1064; in part by the Fundamental Research Funds for the Beijing University of Posts and Telccommunications under Grant 2025A14S02.
References
[1] Eneko Agirre, Llu'is M M'arquez, and Richard Wicentowski, editors. Proceedings of the Fourth In- ternational Workshop on Semantic Evaluations [SemEval-2007) Association for Computational Linguistics, Prague, Czech Republic, June 2007.
[2] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, et al. Falcon-40b: an open large language model with state-of-ther thethha-tt performance, 2023.
[3] Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Vic- toria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language modeling with mixtures of experts. arXiv preprint 1rXiv:2112. 2021.
[4] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint
2021.
10

---

[5] Baidu-ERNIE- ERN-E-TTaIEE.. Ernie 4.5 technical report. 2025.
[6] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint M.Xiv:2401.0295:: 2024.
[7] Weilin Cai, Juyong Jiang. Le Qin, Junwei Cui, Sunghun Kim, and Jiayi Huang. Shorteut-connected expert parallelism for accelerating mixture-of-expert. arXiv preprint arXiv:2404.05019, 2024.
[8] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey mixture of experts. arXiv preprint arXiv:2407.06204 2024.
[9] Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. survey on mixture of experts in large language models. IEEE Transactions on Knowledge and Data Engineering. 2025.
[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint urXiv:2107.03372 2021.
[11] Tianlong Chen, Zhenyu Zhang, Ajay Kumar Jaiswal, Shiwei Liu, and Zhangyang Wang. Sparse moe as the new dropout: Scaling dense and self-slimmable transformers. ICLR, 2023.
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 10Xiv:2/10.14111 2021.
[13] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177-190. Springer, 2006.
[14] Damai Dai, Chengqi Deng, Chenggang Zhao, Rx Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng. Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in oixture-of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), pages 1280-1297, 2024.
[15] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344-16359, 2022.
[16] DeepSeek-AI. Deepseek-v2: A strong, economical, and efficient mixture-of-exxpert language model, 2024.
[17] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in neural information processing systems, 36: 10088- 10115, 2023.
[18] William B Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In Proceedings of the International Workshop on Paraphrasing, 2005.
[19] William Fedus, Jeff Dean, and Barret Zoph. review of sparse expert models in deep learning. arXiv preprint 27X1v:2209.016667 2022.
[20] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23 (120):1-39, 2022.
[21] Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-loras: efficient multitask tuning for large language models. arXiv preprint arXiv:2403.0343) 2024.
11

---

[22] Chongyang Gao, Kezhen Chen, Jinmeng Rao, Ruibo Liu, Baochen Sun, Yawen Zhang, Daiyi Peng, Xiaoyuan Guo, and VS Subrahmanian. Mola: Moe lora with layer- expert allocation. In Findings of the Association for Computational Linguistics: NAACI. 2025, pages 5097-5112,
2025.
[23] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pages 1-9. Association for Computational Linguistics, 2007.
[24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407. 2024.
[25] Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, and Tao Lin. Dynamic mixture of experts: An auto-tuning approach for efficient transformer models. CoRR, abs/2405.14297, 2024.
[26] Xumeng Han, Longhui Wei, Zhiyang Dou, Zipeng Wang, Chenhui Qiang, Xin He, Yingfei Sun, Zhenjun Han, and Qi Tian. Vimoe: An empirical study of designing vision mixture-of-expert.. arXiv preprint arXiv:2410.1577 2024.
[27] Xin He, Shunkang Zhang, Yuxin Wang. Haiyan Yin, Zihao Zeng, Shaohuai Shi, Zhenheng Tang. Xiaowen Chu, Ivor Tsang, and Ong Yew Soon. Expertflow: Optimized expert activation and token allocation for efficient mixture-of-experts inference. arXiv preprint 1rXiv:2410.17954 2024.
[28] Ahmed Hendawy, Jan Peters, and Carlo D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. arXiv preprint arXiv:2311.11385, 2023.
[29] Ahmed Hendawy, Jan Peters, and Carlo D' D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Represen- tations, 2024.
[30] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[31] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[32] Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder task needs more experts: Dynamic routing in moe models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12883-12895, 2024.
[33] Yongqi Huang, Peng Ye, Chenyu Huang, Jianjian Cao, Lin Zhang, Baopu Li, Gang Yu, and Tao Chen. Ders: Towards extremely efficient upcycled mixture-of-experts models. arXiv preprint arXiv:2503.01359, 2025.
[34] Ranggi Hwang, Jianyu Wei, Shijie Cao, Changho Hwang, Xiaohu Tang, Ting Cao, and Mao Yang. Pre-gated moe: An algorithm-system co-design for fast and scalable mixture-of-exper inference. 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pages 1018-1031. IEEE, 2024.
[35] Gagan Jain, Nidhi Hegde, Aditya Kusupati, Arsha Nagrani, Shyamal Buch, Prateek Jain, Anurag Arnab, and Sujoy Paul. Mixture of nested experts: Adaptive processing of visual tokens. The Tiihty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[36] Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim, Muhammad Abdul- Mageed, Laks VS Lakshmanan, Ahmed Hassan Awadallah, Sébastien Bubeck, and Jianfeng Gao. Automoe: Heterogeneous mixture-of-expprts with adaptive computation for efficient. neural machine translation. In ACL (Findings), 2023.
12

---

[37] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 240.0008000000888 2024.
[38] Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts. arXiv preprint arXiv:2406. 12200.12004, 12034, 2024.
[39] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
[40] Hector J Levesque, Ernest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, page 47, 2011.
[41] Jia LI, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Li Zhou, Yann Fleureau. Guillaume Lample. and Stanislas Polu. Numinamath. wOw./AI-NO////tiiaaaaaaao------ project- prw.iiiiiiiooooooreeeeeeeeeeeeea wrwppppiiiaooee /tt.a///oooooggeessrrrr/a///a/aa 2024.
(uttps://githann com/ http:/etadddattt.p.aaaade. pdf),
[42] Jing Li, Zhijie Sun, Xuan He, Li Zeng. Yi Lin, Entong Li, Binfan Zheng, Rongqian Zhao, and Xin Chen. Locmoe: A low-overhead moe for large language model training. arXiv preprint arXiv:2401.13920. 2024.
[43] Jing Li, Zhijie Sun, Dachao Lin, Xuan He, Yi Lin, Binfan Zheng, Li Zeng, Rongqian Zhao, and Xin Chen. Expert-token resonance: Redefining moe routing through affinity-driven active selection. arXiv preprint arXiv:2406.000033 2024.
[44] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint 2rXiv:2305.20050. 2023.
[45] Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang. Munan Ning, et al. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401. 15947, 2024.
[46] Aixin Liu, Bei Feng. Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseck-v3 technical report. arXiv preprins arXiv:2412.0 19437, 2024.
[47] Boan Liu, Liang Ding, Li Shen, Keqin Peng, Yu Cao, Dazhao Cheng, and Dacheng Tao. Diver- sifying the mixture-of-experts representation for language models with orthogonal optimizer. arXiv preprint arXhn:2310.09762 2023.
[48] Jingyuan Liu, Jianlin Su, Xingcheng Yao, Zhejun Jiang, Guokun Lai, Yulun Du, Yidao Qin, Weixin Xu, Enzhe Lu, Junjie Yan, Yanru Chen, Huabin Zheng, Yibo Liu, Shaowei Liu, Bohong Yin, Weiran He, Han Zhu, Yuzhi Wang. Jianzhou Wang, Mengnan Dong, Zheng Zhang, Yongsheng Kang, Hao Zhang, Xinran Xu, Yutao Zhang, Yuxin Wu, Xinyu Zhou, and Zhilin Yang. Muon is scalable for IIm training, 2025. URL https: / arxivv. org/abs 2502. 16982.
[49] Xinyi Liu, Yujie Wang, Fangcheng Fu, Xupeng Miao, Shenhan Zhu, Xiaonan Nie, and Bin CUL Netmoe: Accelerating moe training through dynamic sample placement. The Thirteenth International Conference on Learning Representations, 2025.
[50] Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of- experts large language models. arXiv preprint arXiv:2402.I 14800, 2024.
[51] Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large language models. arXiv preprint 12884, 2024.
13

---

[52] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye. Yiming Yang, al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534 2023.
[53] Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, et al. Gemma: Open models based on gemini research and technology. CoRR, 2024.
[54] Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multi- modal contrastive learning with limoe: the language-image mixture of experts. Advances in Neural Information Prucessing Systems, 35:9564 2022.
[55] Nabil Omi, Siddhartha Sen, and Ali Farhadi. Load balancing mixture of experts with similarity preserving routers. arXiv preprint arXiv:2506. 14038, 2025.
[56] Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training, sparse inference: Rethinking training of mixture- of-experts language models. CoRR, 2024.
[57] Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems,
2023.
[58] Peijun Qing. Chongyang Gao, Yefan Zhou, Xingjian Diao, Yaoqing Yang. and Soroush Vosoughi. Alphalora: Assigning lora experts based on layer training quality. In EMNLP, 2024.
[59] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proc progle-prooffe q&a benchmark. In First Conference on Language Modeling, 2024.
[60] Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, al. Mixcure-of-experts meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305. 14705, 2023.
[61] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of EMNLP, pages 1631-1642 2013.
[62] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TRANSACTIONS ON MACHINE LEARNING RESEARCH, 2022.
[63] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether arXiv:2210.09261, 2022.
can solve them. arXiv preprint
[64] Peng Tang, Jiacheng Liu, Xiaofeng Hou, Yifei Pu, Jing Wang, Pheng-Ann Heng, Chao Li, and Minyi Guo. Hobbit: mixed precision expert offloading system for fast moe inference. arXiv preprint arXiv:2411.01433. 2024.
[65] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[66] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint
2018.
[67] Kun Wang, Guibin Zhang. Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. comprehensive survey in Im (-agent) full stack safety: Data, training and deployment. arXiv preprint 2rXiv:2504.15588 2025.
14

---

[68] Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxiliary-loss-fii fress-freeeeeeee load balancing strategy for mixture-of-experts. arXiv preprint arXiv:2408. 15664, 2024.
[69] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484-13508, 2023.
[70] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint (rXiv:2406.01574 2024.
[71] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. arXiv preprint 1805, 2018.
[72] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners, 2022. org/ 22009. 01652.
URL https:
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. 2022.
Advances in neural information processing systems,
[74] Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, et al. Symbol tuning improves in-context learning in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 968-979, 2023.
[75] Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. Skywork-moe: A deep dive into training techniques for mixture--f-expertt language models. arXiv preprint arXiv:2406.0633
2024.
[76] Colin White. Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-fee LLM benchmark. In The Thirteenth International Conference on Learning Representations, 2025.
[77] Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACI-HHLT,, 2018.
[78] Qiong Wu, Zhaoxi Ke, Yiyi Zhou, Xiaoshuai Sun, and Rongrong Ji. Routing experts: Learning to route dynamic experts existing multi-modal large language models. The Thirteenth International Conference on Learning Representutions, 2025.
[79] Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixuure-ofeexprrs language models. arXiv preprint arXiv:2402.01739, 2024.
[80] Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Lijie Hu, and Di Wang. Moral: Moe augmented lora for Ilms' lifelong learning. arXiv preprint 24Xh::2402.11260 2024.
[81] Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. Adamoe: Token- adaptive routing with null experts for mixture-of mixture-offexpert.. experesss language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6223 6235, 2024.
[82] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, et al. -fxture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
[83] Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moe: Building mixture-of-experts from lama with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Langugge Processing, pages 15913- 15923, 2024.
15

---

[84] Yun Zhu, Nevan Wichers, Chu-Cheng Lin, Xinyi Wang, Tianlong Chen, Lei Shu, Han Lu, Canoee Liu, Liangchen Luo, Jindong Chen, al. Sira: Sparse mixture of low rank adaptation. arXiv preprini arXiv:23/1.09177 2023.
[85] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint
2022.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In both the abstract and the introduction, clearly present the key contribu- tions our paper, including our optimization method based on expert specialization.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide a thorough discussion of the limitations of our work and suggest potential directions for future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: In this paper, we provide the full set of assumption and a complete proof in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In this paper, the experimental code and datasets will be publicly available in the future. The details necessary for reproducing all reported results are thoroughly described in Section 4.2 (Implementation Details).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code and datasets will be publicly released in the future, all reported results are fully reproducible based on the provided data and the detailed implementation described in Section 4.2. Further experimental procedures are documented in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.cc/ppppp wttppcccppppcccccccccccccccccccc mmmblic/guudds: /tt/gguddd///oduuuniiiiiiiii
CoodeSummii issionni ssssssoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the dataset construction process and all experimental details, including hyperparameter settings and other implementation specifics, the Appendix and in Section 4.2 (Implementation Details).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: The vast majority Of experiments in this article report variance measurements.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? [Yes]
Question:
Justification: We report the resource consumption metries for all experimental procedures conducted in this study.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: htthiiettttt/////eeeeeee http::::///euriss..o. httprr/p.....ppbbpc/ccccccceeeae
Answer: [Yes]
Justification: All aspects of this work are in full compliance with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the limitation&&ddiccssionn section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work does not involve high-risk models or datasets; therefore, no addi- tional release safeguards necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all creators and original owners of assets used in this paper (e.g., code, data, models) are properly credited. Furthermore, all relevant licenses and terms of use are explicitly stated and fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, all new assets introduced in this paper thoroughly documented, with corresponding documentation provided alongside them to ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing experiments or research with human subjects; therefore, such details are not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This study did not involve human participants; therefore, no risks, disclosures, Institutional Review Board (IRB) approvals were required obtained.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: The use of large language models is described in detail in both the main text and the appendix.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / www.e///eeeeee/e///e//ce/ /LLM) for what should or should not be described.
23

---

A Notations
Table 2: Notations and Definitions
Notation
Definition
C
Total loss function.
L
Primary task loss.
Louz Co
Auxiliary loss function.
Orthogonality loss.
Variance loss.
A d-dimensional input token vector, z; Rd. Number of tokens in sequence or batch.
N
Ej
The j-th expert network.
gg,
Parameters of the j-th expert network Ej.
Vector of logits output by the routing network for token Tt, h(x;) E IR"".
h(x;))
The j-th component of the logit vector h(x;)) corresponding to expert j.
P(ad)s
Initial routing probability token r; for expert E,.
T
Set of indices of the top k experts selected for token æq. Final routing weight assigned to the i-th token for the j-th expert.
si
yi
Final output for token x; from the MoE layer.
Ej (z()
Output of expert E, for token 2;.
fi
Proportion of tokens assigned to expert j.
Sum of routing probabilities (scores) assigned to expert j
Pj
all N tokens in batch, Pj Di=1 Sij-
Indicator function ensuring ž.j is (x; if siz > gate
and zero otherwise.
OR
Parameters of the routing network. Raw logit produced by the routing network for token an; and expert
(i)(OR)
Soft routing probabilities obtained via softmax function applied to logits (0R))
s'yj
Approximate average output of experts for token x; when experts become similar.
(x)g(I.)
Output of expert E, for token 2; Siy > Tgates zero vector otherwise; ë.j (z;) [s:y><guce>
Zij
Threshold for routing score $ij to consider an expert active for orthogonality loss calculation.
Tgate
Small constant added to the denominator in orthogonality loss prevent division by zero. Vector projection of Øig onto Pik-
Gnorm
projan (Zig)
B Motivation
B.1
MoE Layer Structure
Mixture of Experts (MoE) layer enhances the capacity of neural network model by conditionally activating different specialized sub-networks, known as "experts," for different input tokens. This architecture allows the model to scale its parameter count significantly while maintaining a relatively constant computational cost per token during inference.
Let the input to the MoE layer be sequence of N tokens, denoted as X (x1, (01, ..,.... ZN }, where each token æ; E IRd is dddimenssionl vector. The MoE layer comprises set of independent expert networks, F. = (F,, Fe.... F.). Each expert F. is typically feed-forward network (FFN) with its own set of parameters Pg,,
24

---

A crucial component of the MoE layer is the routing network, also known as the gating network, G. The routing network takes an input token z; and determines which experts should process this token. It outputs vector of logits h(x;;) E RR, where each component h((2)); corresponds to the j-th expert. These logits are then typically passed through softmax function to produce initial routing probabilities or scores:
exp(h((x))))
P(xi); E exp(h(xa))))
for j ,....,...
(11)
These probabilities P(a:); represent the initial affinity of token x; for expert Eg..
To manage computational cost and encourage specialization, top-k selection mechanism is often employed. For each token Zi, the top experts (where k: < n, often k: = or k = 2) with the highest routing probabilities P(zz) are chosen. Let T; C ( ,n. be the set of indices of the top experts selected for token Tz. The routing scores are then re-normalized or directly used based on this selection. The routing matrix S of dimensions N x captures these assignments:
$11
$12
81n
$21
822
S2n
S
(12)
8N1
SN2
N
where Sij represents the final weight assigned to the i-th token for the j-th expert. If expert is among the top l selected for token v; (i.e., j E Ti), then 8iy is typically derived from P(xi); (e.g.. by re-normalizing the top-k probabilities so they If expert j is not selected for token x; (i.e..) j Ti), then 8ú 0. Consequently, for each token x., the sum of its routing scores across all experts is normalized:
simply 8i P(z:),// Σier, P(ax)))
Bij 1, for = 1,2,....,
(13)
is important note that with a top-k mechanism where < n, most 8ij values for given will be zero.
Each token as is then processed by its selected experts. The output of expert Eg for token my is denoted as Ey (2x)) The final output y, for token v, from the MoF layer is a weighted sum of the outputs from all experts, using the routing scores weights:
V: Σ Ex.))
(14)
Since 8ij = O for non-selected experts, this sum is effectively only over the top l chosen experts for token 24.
To encourage a balanced load across the experts and prevent a situation where only a few experts are consistently chosen (expert starvation), an auxiliary loss function, Lanar, is commonly introduced. Let F = {/1, ./.,..,.. fn represent the proportion of tokens assigned to each expert. More precisely, S can be defined as the fraction of tokens in batch for which expert is among the top k: selected experts, or it can be softer measure, For given MoE layer, the total loss function L consists of two main parts: the primary task loss Ca (e.g., cross-entropy loss in language modeling) and the auxiliary loss Laux:
L= Cr. + Lauz.
(15)
Here, La is computed based on the final output Y {y1, V2. ..... UN } of the MoE layer (and subsequent layers), and a is a scalar hyperparameter that controls the importance of the auxiliary loss term. The auxiliary loss is often designed penalize imbalance in the distribution of tokens to experts. A common formulation for Lour, as referenced in the original text, involves the sum of routing scores per expert:
Loux
(load, importance,).
(16)
25

---

where load, is related to the number of tokens routed to expert j. and importance, is related to the routing probabilities for expert j. Let PJ represent the sum of routing probabilities (scores) assigned to expert j across all N tokens in the batch:
PJ Σ 8ij
(17)
This P3 value gives an indication of the "total routing score" directed towards expert j. The term f3 in the original formulation, representing the proportion of tokens assigned to expert j. can be considered as the average routing probability for expert j over the batch, i.e., Jj = = Σz T(j I(()) is the indicator function, or softer version using Siz. The specific form Laus = Eن f3.P) Py as given in the prompt, if fj is interpreted as an average probability or fraction of tokens assigned, and Pj is the sum of probabilities, then f Py would be ( Σ: - s.y)) However, a more standard auxiliary loss aims to balance the load, often by taking the form of the dot product of the vector of
Ti), where
the fraction of tokens dispatched to each expert and the vector of the fraction of router probability
dispatched to each expert, scaled by the number of experts. For example, a common auxiliary load balancing loss used literature (e.g., Switch Transformers) is:
N I 1=1
N { I 1=1
TÚ (EEE))
P(x.);)
Laur a
(18)
N
or using 8ij values directly related to P(zz); for the selected experts. The intent is to make the product of the actual load (how many tokens an expert gets) and the routing confidence for that expert more uniform across experts. If fj in the original text refers to N,/N (fraction of tokens routed to expert j) and pj is Σ Sty (sum of gating values for expert j over the batch, which already considers the top-k selection implicitly through sty), then the formula from the prompt:
L= +
f
(19)
where fj is the proportion of tokens assigned to expert j. and p; = E Sij is the sum of routing weights for expert j. This auxiliary loss encourages the gating network to distribute tokens such that experts with higher Py (receiving larger aggregate routing weights) are also assigned a substantial fraction of tokens fy. aiming for balance in expert utilization.
B.2 Observation
Obs 1(Expert Overlap): Introduction of the auxiliary loss function leads more homogenized distribution of tokens across experts, which may reduce the distinctiveness of each expert.
It has been observed that the auxiliary loss function is independent of the expert parameter matrices 0 Dz Therefore, for the j-th expert, its gradient can be written as:
OC aCn a0R, 00R, F, 00F,
dLaux
OC dyh Dyn 0
N
+
Σ
24 Sij. = 1,2, n.
(20)
where Oz, is the parameter matrix of the j-th expert, and yb is the output of the MoF layer. During gradient descent, the addition of the auxiliary loss Loaz forces the routing mechanism evenly distribute the tokens across experts as much as possible. This results in input token x; being assigned to an expert that may not be semantically aligned with it, causing an unintended gradient flow to expert j. Mathematically, after applying the top-k mechanism, the routing score Sij transitions from 0 to a non-zero value, introducing gradients from tokens that originally had no affinity with expert j.
Obs H(ouuting Uniformity): As training progresses, the routing output tends to become more uniform, with the expert weight distribution gradually converging lowards an equal allocation.
To understand this phenomenon, we first examine the source of gradients with respect to the routing parameters Qg. Let Way (0R) denote the raw logit produced by the routing network for token z; and
26

---

expert j. The soft routing probabilities, denoted as s'y. are typically obtained via a softmax function
applied to these logits:
cxp (0x))
5/1
(21)
Ri=i exp (0R))
These soft probabilities s'y are then used determine the final routing assignments Sty in the matrix
S (after top-k selection). The derivatives Øssi in the gradient expressions are understood to represent the differentiation through these underlying soft probabilities with respect to the router parameters 0R. The total loss C comprises the main task loss Ln, and the auxiliary loss Loux. The gradient of L with respect to 0 is given by:
ac
dLn
+ a HLaux
(22)
00R
00R
00R
Substituting the expressions provided in the context, have:
ac
N
Oesi Nnn
8ss
Σ i=1
Qgz)
+
(23)
n
00n
where 24 SE, represents the output of expert for token Ii, and fj denotes the fraction of tokens ultimately assigned to expert j.
The first term, 30
Jsy. J0p represents the gradient contribution from the main task loss. This term guides the router to select experts that are most beneficial for minimizing La- However, discussed in Obs I, the expert parameters 0g, tend to become similar during training due to overlapping token assignments induced by Laux. Consequently, the expert outputs σ; OF, become less distinguishable across different experts j for a given token zy. Let σ3 0gg z Eavg for all j. In this scenario, the specific choice of expert j (i.e., making Siz large for that j) has progressively similar impact on C,. regardless of which j is chosen. The differential information (2t, 0g ) (s; (g.) ) between experts diminishes. As result, the router receives weaker, less discriminative signal from the main loss component for selecting specific experts. The ability of L.. to guide fine-grained, specialized routing decisions is therefore reduced.
aCn = E EE= 2z
With the diminishing influence of aCn, the updates to the routing parameters 0₂ become increasingly 09p'
dominated by the auxiliary loss gradient, a DCazz:
80n
ac
dLaur 00R
Esy 00 R
8
(24)
00 R
The auxiliary loss Loux is designed to encourage balanced load across experts, primarily by promoting uniformity in Jj (the fraction of tokens processed by expert j). This is achieved by using P, =m= Say (where si are the post-top-k scores) differentiable surrogate guide the optimization. The objective is to drive f, 1/n for all experts. The gradient term @hams adjusts the router parameters oR (and thus the soft probabilities s' which determine 8:y) achieve this balanced distribution.
89n
In the absence of strong, discriminative signals from La (due expert similarity), and under the primary influence of Louz which penalizes load imbalance, the router tends to adopt a strategy that most straightforwardly achieves load balance. This often results in the soft routing probabilities s/y for given token z; becoming more uniform across the experts j. i.c., sty + 1/n. If the router assigns nearly equal soft probabilities all experts for any given token, then the post-top-k scores Si will also reflect this reduced selectivity, and the sum P; = Σ, Si will naturally tend towards N/n, satisfying the auxiliary loss's objective. This trend leads to the variance of routing weights for given token 2; (i.e., Var,(s',)) decreasing over time. Consequently, the overall routing output becomes more uniform, and the router becomes less specialized its assignments, reinforcing the homogenization observed. This feedback loop, where expert similarity weakens task-specific routing signals and strengthens the homogenizing effect of the load balancing mechanism, explains the progressive trend towards routing uniformity.
27

---

C
Method
C.1
Specialized Losses Co and Cu
In this section, we introduce two critical loss functions: the orthogonality loss Co. which acts on the expert representations. and the variance loss L,, which acts on the routing scores. These losses are designed to encourage expert specialization and routing diversity, respectively.
Expert Specialization via Orthogonality Loss Lo the representations learned by different experts for the same input token as independent as possible. Orthogonal vectors are the epitome of linear independence. Thus, we introduce orthogonality objective that penalizes similarities between the output representations of different experts when they selected process the same token. This is achieved through the orthogonality loss Lo.
To foster expert specialization, we aim make
The orthogonality loss is defined as:
n
(Txys 3ick) (Tsk, Fik) + 6norm
L.
Tike
where
ブúj = Ex(=)) [ooy)
(25)
Here, N' is the number of tokens in a batch, and n is the total number of experts. The input token is denoted by x;. The term Ey(x:) represents the output vector of the j-th expert, E,, when processing token 24. The indicator function T( (Tate) ensures that Ziy is the actual output E, (ax) if the routing score 8tj for expert and token 2; exceeds certain threshold Taate (implying expert j is selected in the top-k routing for token x.), and Tig is a zero vector otherwise. This effectively means that the loss operates only on the experts that are active for a given token. A small constant cnorm is added to the denominator to prevent division by zero if an expert's output vector happens to be zero.
Fa)+ En.. calculates the vector projection of the output ë.y (from expert for token i) onto the output Tyh. (from expert lo for the same token i). The loss Co sums these projection vectors for all distinct pairs of active experts (j. k-) for each token Ty, and then sums these across all tokens in the batch.
The core component of Co. proj a (Zay)
(Bs).aa)
Although the formula (25) presents f. as sum of vectors, the optimization objective is to minimize the magnitude of these projection components. Typically, this is achieved by minimizing scalar value derived from these vectors, such as the sum of their squared L2 norms, i.e., Eijj,yj ||proj a. (Txz) 1 Minimizing these projections encourages the dot product (āij, Task) to approach zero for j * k. This forces the representations ēey and R... from different active experts to become more orthogonal.
By minimizing for we reduce the representational overlap between different experts chosen for the same token. This encourages each expert to learn unique features or specialize in processing different aspects of the input data, leading to a more diverse and efficient set of experts. This specialization is key to mitigating the expert overlap issue.
Routing Diversification via Variance Loss L. To ensure that the router utilizes experts in a varied and balanced manner, rather than consistently favoring a few, we introduce variance-basee loss L.. This loss encourages the routing scores assigned by the router to be more diverse across tokens for any given expert.
The variance loss is defined
N (sij - 3,) where 3j
n
N SK
-- -
(26)
n2
N
In this formula, Siy represents the routing score (e.g.. gating value from a softmax layer in the router) indicating the router's preference for assigning token 2; expert Ej. The term 8j is the average routing score for expert F, calculated across all N tokens in the current batch. This average score, sy. can be interpreted as a measure of the current utilization or overall assignment strength for expert j within that batch.
The core of the loss, (sij - 3,)2, measures the squared deviation of the specific score Say from the average score 3j for expert j. A sum of these squared deviations for a particular expert j over all tokens, E quantifies the total variance of routing scores received by that expert. A
28

---

high variance implies that expert ý receives a wide range of scores from different tokens (i.e., it is strongly preferred for some tokens and weakly for others). rather than receiving similar scores for all tokens it processes.
The loss L. sums these squared deviations over all experts j (scaled by 1/n) and all tokens i, and then negates this sum. Therefore, minimizing Cu is equivalent to maximizing the sum of these score
variances: プド= N ( 8,)2. This maximization encourages the routing mechanism to produce a diverse set of scores for each expert across different tokens.
By promoting higher variance in routing scores per expert, Le helps prevent routing uniformity, where experts might be selected with similar probabilities for many tokens or where some experts are consistently overloaded while others are underutilized based uniform high/low scores. Instead, it pushes the router to make more discriminative assignments, which can lead to better load balancing in conjunction with Laua and supports experts in specializing on more distinct subsets of tokens.
C.2 Compatibility of Multi-Objective Optimization
In this section, we conduct detailed analysis of how each loss component, namely Lhs Launs Las Lev influences the optimization dynamies of expert parameters 0F, (for = .,...." experts) and routing parameters Or during the training process. Our primary focus to demonstrate the theoretical com- patibility and synergistic interplay between the specialized losses Co (promoting expert orthogonality) and C, (promoting routing score diversification) in conjunction with the load balancing loss Loux and the primary task loss Li. The analysis is structured around two key questions:
Balancing Expert and Routing. How can expert (Lo) and routing ( (Le) optimizations be designed to complement each other without compromising their respective objectives, and how do they interact with Caun?
We begin by demonstrating that the optimization objectives C. and Lu are compatible in their optimization directions with respect the expert parameters OF, and routing parameters 0z- Subse- quently, we will show that these losses can mutually reinforce each other, leading to more effective and stable learning process for Mixture-of-Expert. (MoE) models.
Mutually Compatible
We elaborate on the compatibility of L. and Le by examining their respective gradient contributions to expert parameters and routing parameters. The total loss function is L L, +aLouz + 3L, +Y
From the expert parameter 0F perspective, the expert parameters 0r, are primarily updated to minimize the task loss La for the tokens routed to expert j. and satisfy the orthogonality constraint f.g- The auxiliary loss Lava and the variance loss L, are functions of the routing scores 8ij (outputs 8Caus = 0
of the router R(x:)o.)) and do not explicitly depend on the expert parameters ,g,. That is, 00g,
and ac_ 0. The output of expert for token z; is denoted as ã., E, (z;; (g,) I[s,y20) where 00g
the transformation by expert (e.g.. 2,0g if x; is row vector and 0g, is a weight matrix), and [(s,>>0) is an indicator function that is if x; is routed to expert j (i.c., Sij is among the top-k scores for 2;)) and 0 otherwise. For simplicity in gradient derivation with respect to 6g we consider only tokens x; for which expert is active. The gradient of the total loss L with respect to FE is:
Ac
N Σ
0hn Dā.
+ 3 aLo
0ū:j 00g,
[180200
(27)
00E,
dī,;
Let In: By. &Cn Siy- The orthogonality loss L. is designed to make ž. and ã. (for + j. also selected Bag
aCs. be the gradient of the task loss with respect to the final output yı = E. T. Then
for xz)) orthogonal. Assuming the specific form of L, from the paper leads to the gradient component shown (interpreted as 0ā,y
& ac. contributing Σn Ek=1keg (R.)) Piy). ). and dū; results in factor of x: a0s,
29

---

(assuming Zi = θg, 24 with 2; as column vector), the gradient expression given in the paper is:
0C
N
Tu Tegs (Tike Tike
Σ
203
81.8ij +
x..
(28)
1=1
More generally, using the paper's notation for the gradient w.r.t. DE, directly:
N
OC
Σ
gCn (Ztr, (6,) ) +B gc. gc. (1 Ris)
(29)
00 E,
00g,
from
from C.
where gc, (Faj. Sty) represents the gradient contribution from La to Fig (e.g., 8új in the paper's simplified notation might represent Si: similar term) and Ic. ( h Fiz) represents the
gradient contribution from Lo (e.g.. Σ Byclie Biy if it acts on Tzy). The crucial observation is
kgj
that Lour and L. do not directly impose conflicting gradient directions on 0E, as their influence is on 8g. As training progresses, Lo encourages 8g, to form specialized representations. This specialization, driven by Lo, is not hindered by Lauz or L.
From the routing parameter R perspective, the routing parameters 0 determine the routing scores Sij R(I;; On). The gradient of the total loss with respect On is given by:
Oc 00R
ΣΣ OC 0súj
(30)
Dsij 00
The term
ac captures influences from all relevant loss components:
dag
ac
aCh
DLouz
OC.
ac.
(31)
Us;;
Us.j
Os.j
Östj
The paper asserts that Co does not directly affect the gradient with respect to routing parameters 0g. implying ac_ = 0. This holds if L, is defined based on the expert outputs žig which, once an expert deg
is selected, depend 0g, and x; but not the magnitude of sty itself (assuming 8, is used for hard selection via top-k, and not as a differentiable weighting for Tig within C. 's definition). Given this assumption, the gradient Da13
becomes:
ac
2Lax dsij
OC. Osij
Ty.
Taj +
(32)
Osij
from Ca
from Lus
from Co
Substituting the specific forms for derivatives Laux and L. (where Cano often involves balancing
the load f, = Σ, st//N or similar, and L. = -N E (sig - 82)2), the paper's specific form for the gradient of the total loss w.r.t. OR is:
2c
NN
2(N - 1) nNV
0sg
T
+a derived from fo
(815 - 3,)
(33)
00
00
serm from Cs.
term from Cuur
term from C.
The term represented by žij in the paper's original routing gradient formula corresponds to 9f žij. fy the derivative of Laux. and the last to the derivative of C.. This gradient is influenced by the expert representations žij (via L.), the expert load f; (via Laur), and the distribution of routing weights Siy (via L.). The optimization of C. aims to diversify routing scores, while Lour aims to balance loads. These objectives are not inherently contradictory with the primary task of minimizing Lx. For instance, L. might encourage token to strongly assigned to one expert within its top-k set, while Lauz ensures that, across all tokens, experts are utilized in a balanced manner. The absence of direct gradient from L., on 8iy (and thus 0z) prevents direct conflicts between expert orthogonalization and the routing objectives.
30

---

Based on this detailed analysis of gradient components, we can summarize:
Summary. Expert parameters Og, updated based on gradients from Cn and Co. The losses Lava and C, do not directly contribute gradients to θE,, thus avoiding conflicts with expert specialization. Routing parameters 0g are influenced by gradients from Cr, Lauz, and L.. The objective of Co (expert orthogonality) does not directly impose constraints on 0R. and the objectives of Lauz (load balancing) and L. (score diversification) are designed to be compatible aspects of routing.
Mutually Reinforcing
Beyond mere compatibility, L. and L, can create a synergistic effect, where improvements one facilitate the optimization of the other.
The orthogonality loss L. encourages the effective output vectors of different selected experts, Zy and Ts (for j * k and both j, k selected for token æ2), to become more orthogonal, i.e., (Ztz. Z x O. The learning signal for the routing mechanism, particularly the part derived from the primary task loss Cn with respect to the routing score Sij, is crucial. This component is given by:
ac
dcn dy:
HCh
oh. Ftg. where y Σ Stn and Iv. 0y:
(34)
dsi
dy, Osij
The full gradient for Sıj (excluding C. direct term as discussed) is:
ac
HLaux
0h. Eas
Sy.
+ a
(35)
0syj
Ooyg
from Ci.
from
Let Pi, = =9 Big represent the projection of the task gradient Jy, onto the expert output ø.y. When the expert outputs {āy}} for a given token x; tend to be orthogonal, they represent distinct, non- redundant features. For any given task-specific gradient vector 9y., its projections Pty onto these more orthogonal expert output vectors are likely to exhibit greater variance. For example, if ø.,,, and Rz are orthogonal, 9y. might align well with غi,J (large pi.,) ) but poorly with Ta.ja (small Pi.ja). In contrast, if ãe.j; and ii.j were nearly collinear, Pt.j; and Pi.ja would likely be very similar. This increased variance in the task-relevant signals Pi; provides the routing mechanism with more discriminative information, making it easier to differentiate between the utility of experts for a given token. This, in turn, creates more favorable conditions for L., which aims to maximize the variance of routing scores Sij, thereby encouraging more decisive routing decisions.
Conversely, E. contributes to expert specialization. By promoting diverse routing scores Sty, Cy en- courages the router to send different types of tokens different experts (or to assign tokens with higher confidence to a smaller subset of the top-k experts). This results in each expert E, being trained on a more specialized subset of tokens, denoted T {a; expert j is selected for x; with high score), As experts see more distinct data distributions, their parameters fg, are more likely to diverge and learn unique features representative of their assigned token subsets T,. This functional divergence naturally promotes the orthogonality of their output representations æ.j. which is the direct objective of L.. Thus, C, indirectly aids C.. The statement the original text "due to the influence L, gradient &C. on fn" interpreted here as an indirect influence: C. improves expert representations Taj, which in turn makes the routing signal o Styj more discriminative, thereby influencing On-
Baig
Summary. A virtuous cycle is formed: Lo promotes orthogonal expert outputs žıj, which enhances the discriminative power of the routing signals S" ãzy. More discriminative routing signals allow L. more effectively diversify routing scores 8g.. In turn, diversified routing scores S. driven by L. lead to experts being trained on more specialized token subsets, which facilitates the learning of divergent and orthogonal expert parameters .g,, thus supporting the objective of L.- This mutual reinforcement contributes to overall model stability and performance.
Multi-Ohjectice Optimization. How do expert and routing maintain their balance while enhancing Laaz and Ln independently, ensuring mutually beneficial performance improvements?
The overall objective function C aims to optimize four key aspects:
31

---

1. Accurate data fitting and task performance (minimizing Lx).
2. Orthogonal and specialized expert representations (minimizing Co).
3. Balanced load distribution across experts (minimizing Lauz).
4. Diverse and confident routing decisions (maximizing variance via fays i.e., minimizing negative variance).
Our core objective is to achieve an optimal balance by jointly optimizing these multiple objectives, ensuring they complement each other for enhanced model performance. The compatibility of these objectives is supported by the following considerations, including the provided lemmata.
Lemma Let S E xn be the matrix of routing scores, where Sij is the score for token i assigned expert j. Assume for each token x; (row of S), Σド= Sij (if scores are normalized probabilities post-softmax) or that experts are chosen (e.g., Stj E 10,1 or general Sij for selected experts). Then, there always exists a state where the following two objectives are simultaneously optimized: 1. Load balancing: The sum of scores for each experi (column sum, I3 >i= 8iÿ) lends towards an average value, e.g., N. N.k/n k/nn if each token selects experts, or N/n if 8ij are probubilities and 1. This is driven by Laux- 2. Routing score variance: For each token 2ú, the variance of its non-zero routing scores Sty (among the chosen top-k experts) is increased. This is driven by L..
Lemma I suggests that the goals of Laur and C. are not inherently contradictory. Caue focuses on inter-expert load distribution (column- -oise wise property of S), while L, focuses on the concentration of routing scores for each token (row-wise property of S). For example, even if each token z; strongly prefers one expert over others in its top-k set (high variance for st.), the assignment of tokens to experts can still be managed such that overall expert utilization is balanced. Different tokens can strongly prefer different experts, allowing column sums to balance out.
For two objectives, such as (A) making expert representations orthogonal and (B) bal- ancing the computational load across these experts, it is possible to achieve both. If we consider experts needing to learn distinct "regions" of the problem space (orthogonality) and also needing to process a fair share of the data (load balancing). The original phrasing was: "For two sets of points A and B of equal size, it is always possible partition A J B such that A An B Ø and | [a|=Bii -B)." Interpreeed in our context: Let F; be the functional space or feature set that expert for j + k (orthogonaiyyApecciazzziio the computational load on expert j. Louz aims to make C, zs Cs. It is possible for experts to learn distinct specializations (F, are disjoint) while still processing a comparable amount of data or tokens (C₃ are balanced), provided the data itself contains enough variety to be beneficially partitioned among specialized experts.
Lemma 2
specializes in. L. aims to make F,n F3
Let C₁ be
Lemma 2, under this interpretation, suggests that the objectives of expert orthogonalization (Co) and load balancing (Lauz) are compatible. Expert specialization does not necessitate load imbalance, nor does load balance prevent experts from specializing. Each expert can become highly specialized in processing certain types of inputs learning specific features, while the routing mechanism ensures that the number of tokens processed by each expert remains roughly equal.
In summary, the multi-objective optimization framework is designed for compatibility:
Laux and L. can be jointly optimized as per Lemma 1.
L. (leading to expert specialization) and Lava (load balancing) are compatible as per Lemma 2's interpretation.
As discussed the "Mutually Reinforcing" section, L. and C. can synergistically enhance each other. f. makes expert outputs more distinct, which helps L. by providing clearer signals for routing diversification. Diversified routing, in turn, provides more specialized data streams to experts, aiding L..
All these objectives serve to improve the primary task performance Ln. Specialized experts (L.)) can model complex functions more effectively. Balanced load (Lauz) ensures efficient use of resources and prevents undertraining of some experts. Diverse and confident routing (C.)) ensures that tokens are sent to the most appropriate experts.
32

---

This comprehensive approach allows the model to harness the benefits of MoE architectures by promoting expert specialization, efficient resource utilization, and decisive routing, all contributing to better overall performance on the downstream task.
C.3 Proof of Lemmas
to I, each row contains i non-zero elements and zero elements. Then, there always exists a which the following two objectives are simultaneousty optimized: 1. The sum of the elements in each column tends to the average value N
Lemma Let S E R be matrix that satisfies following conditions: each
n 2. The variance of the non-zero elements in each now increases.
proof C.1 1. Preliminaries and Assumptions
The lemma implicitly requires 2 2. If 1, each row has a single non-zero element $t,;
The set of non-zero elements for row is {1}. Its mean and its variance is $ {1 1)² = =0. This variance cannot be increased 8i,; must remain 1. Henceforth, we assume k: 2 2.
Let P = (pes) E {0,1)Xx xx denote the support matrix where pij = if 8ij # 0 and Pij = 0 otherwise. Condition (ii) implies E}=1 for all i.
Construction of an Initial State ((0) Optimizing Objective 1
To optimize Objective I, we select support matrix P such that its column sums (degrees of column nodes in the associated bipartite graph), dj =->- Pij. uniform possible. That is, each dj E [[Nk/m], [Nk/n]]] The existence of such matrix P a known result in combinatorics (e.g.. provable via network flow arguments related the existence of (0, 1)-matrices with given marginal sums).
Define an initial matrix (!0) = (5/g) based on this P:
(0)
1/k
P
0
if; =
This matrix (00) satisfies:
Row sums: Σラ= 8ig Ejmg=11111 (1/k) = k. (1/k) 1 for all
(0)
Columm sums:
0,00= =I 8i
10) = = =1 (1/k) dgy / Since the integers d,
uniform as possible, the values c00) minimize Σ}= (C₁ N/n)². Thus. S(0) optimizes Objective 1.
Row variance:
For any row i, the non-zero elements are all 1/k. The mean of these non-zero elements is p4 (0) (1/k) E,p, (1/k) 1/k. The variance of these non-zero
elements is Var; (S = Σ Ejpss (0) p 10) E 1/k)²
3. Perturbation via Cycle in the Support Graph Gp
Let Gp = (UUV, Ep) be the bipartite graph associated with P, where U (r1....,7 TN) represents rows, V = [cI,.... Cn. represents columns, and an edge (r., (r,, E Ep if and only if Pij = 1.
We assume that for k 2 2, the graph Gp (corresponding described above) contains at least one cycle. If Gp were a forest, this specific perturbation method would apply. The strength of the lemma' 's claim ("always exists") suggests that such cycle is indeed available in an appropriately chosen P.
P that optimizes Objective as
Let such a cycle be P (ri C1
TL - TT). The edges edges forming this cycle (0)
correspond matrix entries sr,.c 87..01 872. for simplicity), all of which are equal to L/k.
(0)
Srier (indices are re-labeled for cycle elements
33

---

Define perturbed matrix S' = (5/z) by altering elements along this cycle. Let be scalar such
that 0 < 8 S 1/k.
S F1,c1 = in + 8 = +
+ 1/k
22.01 ST2,01
22 = (0) + d 1/k
F11,CL STr,CL + 5 1/k // + 8
SricE - 6 1/k: 6
Elements s'g not involved in the cycle remain 8ij" Since S 1/k, all Sig 2 0. The number of non-zero elements per row remains k.
Row Sums of S': For any row in the cycle (e.g. rı), two of its elements are modified: srice by +8 and 8 by -8. All other non-zero elements in row T1 are unchanged. Thus,
TL the sum 1 = Ej Sra. 1. This holds for all rows r.,-.., Rows not the cycle are unaffected.
(0)
Column Sums of S': For any column Cg the cycle (e.g., 61). two of its elements are modified: stn.e,, by +6 and $12,0s by -5. This holds for all columns c1...., CL. Columns C₁ - for all j. and Objective remains optimized.
b--5. Thus. the sum Ej S/,c, = =E S
(0) c
the cycle are unaffected. Therefore,
Row Variances in S': Consider row T1- Two of k non-zero elements are now 1/k + and 1/k 8, while the other k - 2 remain 1/k. The mean of non-zero elements in is μí = & ((k 2 + (1/k + 6) + (1/k 5)) = 1/k. The variance of non-zero elements
row T1 is:
Vari (S')
1
(k 2)
1 2
+ (
+6)
-x 1 ) + -8)
-
k
28
+ 52 + +-58)1 = k k
Since 8 > 0 and cycle, their variance of non-zero elements increases from 0 to 25 /k. Rows not in the cycle maintain zero variance. Thus, Objective is achieved as the variance has increased for at least these L rows.
(ar)(8')= 0. Similarly, for all rows ..,...,T involved in the
4. Existence of the Desired State and Conclusion
The construction of S' from ((0) demonstrates that if 2 and the support graph Gp (chosen to optimize Objective 1) contains cycle, then state S' exists satisfying the lemma's conditions. Objective I remains optimized, and Objective 2 is achieved because the variance of non-zero elements in rows participating in the cycle is strictly increased from zero.
Thus, under the stated assumption of cycle existence in an appropriately chosen support graph Gp, the matrix S' is the desired state.
Lemma 2 For two sets of points A and B of equal size, it is always possible to partition A U such that A 1 Ø and A [B].
proof C.2 The lemma aim to prove states: For two sets of points A and B of equal size, it is always possible to partition A such that the components of this partition (also referred to A and B in the conclusion of the lemma) satisfy An B = 2 and AA| BB).
34

---

For the lemma's 's assertion that this is "always possible" to hold, we interpret the conditions "An Ø" and "|A] = \ the conclusion as pertaining to the initially given sets A and B themselves. These sets must satisfy these conditions and thereby form the required partition of their union.
Let A and B be sets of points. From the statement of the lemma and interpretation, establish the following premises:
(i) The sets A and B are of equal size, i.e., there exists non-negative integer such that n...Te (This is given by the lemma's hypothesis.)
A = [B] = n.
(ii) For A and serve as the components of the partition of AUB and to satisfy the disjointness condition in the lemma's 's conclusion, require that the sets A and B themselves are disjoint, i.e., An Ø.
We now demonstrate that under premises (i) and (üi), the sets A and B form partition of their union, A U B. First, consider the pair of sets (A,B) as a candidate partition for the set U AUB. For
(A, B3) to be valid partition of U, its components must satisfy two conditions:
The union of the components must be equal to the being partitioned. Here, AUB by definition equal to U.
The components must be mutually disjoint. By premise (ū), we have An B Ø.
Thus, the sets A and B indeed form valid partition of A.B. B.
Next, we verify that the components of this partition (i.e., A and B) satisfy the specific properties mentioned in the conclusion of the lemma:
1. The components of the partition are disjoint. This condition is An B 2, which is true by premise (ii).
2. The components of the partition are of equal size. This condition is | [B], which true by premise (i).
Since the components of this partition (formed by and B themselves) satisfy all the properties required by the lemma's 's conclusion, the lemma is proven.
C.4 Computational Overhead of Lo
While Lg has quadratic complexity in theory, the actual overhead is negligible in practice due the small number of activated experts (k) and efficient batched implementations. It does not present a bottleneck in our setup. Detailed experimental results are provided in Appendix J.
Lo involves pairwise projections among the & selected expert outputs for each token, leading to theoretical cost of O(N 122.. d). In practice, this cost remains manageable for three reasons.
(1) Small & in standard MoE practice. Sparse MoE models typically keep small control computation. In our experiments, we follow this convention, using configurations such as k = 6 in DeepSeck-V22 VV--Lie. Given that < N and < the quadratic factor contributes minimally the overall training cost.
(2) Efficient hardware execution. The main operations in Lg-inner products and pairwise pro- jections- are highly parallelizable and efficiently implemented as batched matrix multiplications in frameworks such PyTorch, running smoothly modern GPUs.
(3) Justified by empirical gains. The modest increase in computation offset by substantial and consistent performance improvements across diverse downstream tasks. This demonstrates that the regularization effect of Lo leads to meaningful gains in expert specialization without incurring prohibitive cost.
D Datasets
GSM8K [12] a benehmark designed to evaluate mathematical reasoning through 8,000 elementary and middle school word problems across arithmetic, algebra, geometry, and other topics. Each
35

---

problem comes with detailed step-by ste solutions, enabling models to learn chain- of thought (CoT) reasoning strategies. The dataset is widely used train and assess model's ability decompose multi-step questions logically and produce interpretable solutions.
MATH500 [44] focuses advanced mathematics with 500 university-level problems in calculus, linear algebra, abstract algebra, real analysis, and more. Problems typically require multi-step formal proofs, symbolic manipulation, and theoretical understanding, making it a strong test for mathematical maturity. Its emphasis on rigor and abstraction makes it ideal for developing specialized solvers and assessing formal reasoning depth.
Numina [41] is a large-scale math dataset containing approximately 24,000 problems ranging from primary to high school levels, annotated with explicit chain-of- chain-of-thouuhtt reasoning steps. It is designed to teach models to perform structured, stepwise reasoning rather than shorteut memorization of solutions. The dataset is particularly effective for improving multi-step performance and explainability in math-based language models.
MMLU [31, 30] is a massive multitask benchmark with multiple-choice questions spanning 57 academic subjects, including science, humanities, law, and medicine. Each subject is stratified by difficulty (high school to expert level), allowing evaluation across broad spectrum of general knowledge. MMLU is a widely adopted standard for testing recall in large language models (LLMs).
reasoning and factual
MML.Upro [70] an expert-level extension of MMI.U that increases question difficulty by expand- ing answer choices and emphasizing multi-step, high-complexity problems. It targets challenging domains like STEM reasoning and policy analysis, where simple factual recall insufficient. MMLU- pro is ideal for benchmarking models under professional-gaade conditions with nuanced and layered reasoning requirements.
BBH [63] consists of hundreds of diverse tasks covering complex scenarios such as logical reasoning, language games, social sciences, and physical commonsense. Its design aims to challenge models on unconventional capabilities, such as counterfactual reasoning and cross-lingual transfer. Most BBH tasks are open-ended, requiring the integration of commonsense and creative thinking- for example. generating poetry or designing ethical AI frameworks.
GLUE [66, 71, 61, 18, 1, 77, 13, 23, 40] is a foundational NLP benehmark combining nine language understanding tasks such as sentiment classification, sentence similarity, and entailment detection. provides a standardized framework assess general ceneral-purpose language comprehension and model transferability across tasks. GLUE has been instrumental in shaping the early progress and comparison of pre-trained language models.
HumanEval [10] is a code generation benchmark released by OpenAI, containing 164 Python programming tasks with unit test specifications. It focuses assessing model's 's ability synthesize functionally correct, efficient, and stylistically appropriate code from natural language prompts. HumanEval remains key benchmark for evaluating reasoning, planning, and syntax correctness in code generation models.
MBPP [4] features thousands of Python programming problems based on real-world development scenarios like string parsing, API use, and algorithm design. Each task includes input/output specifications and test cases, enabling automated evaluation of code correctness and performance. MBPP is widely used to train and evaluate models for practical software engineering and step-by-step code synthesis.
LiveBench [76] is a real-time evaluation benchmark capturing dynamic user-model interactions from deployment environments like chatbots or decision engines. It tracks response latency, robustness, and contextual consistency streaming or multi-tura settings. LiveBench is designed to reveal edge-case failures and test a model's adaptability under realistic, time-sensitive constraints.
GPQA [59] a high-difficulty multiple-choice dataset written by domain experts in biology, physics, and chemistry, targeting scientific reasoning at an expert level. Questions often require interdisci- plinary integration and reasoning across theory, data interpretation, and experimental design. GPQA is ideal for probing model's capabilities in abstract scientific synthesis and expert-level domain understanding.
36

---

E
Metrics
Max Vax [68] metric introduced quantify load imbalance in Mixture-of-Expere (MoE) models.A lower value indicates more balanced expert utilization, while higher value reflects severe imbalance. evaluates global load balance across the entire validation set, reflecting long-term efficiency and fairness in expert usage.
max; Load, Load, Load,
Maz V Viggloaal
(36)
where:
Load, is the number of tokens assigned to expert
Load, is the average (ideal balanced) load across experts.
Accuracy (ACC) a metric that measures the proportion correct predictions made by a model.It's
calculated as the number of correct predictions divided by the total number of predictions.
Number of Correct Predictions
ACC
(37)
Total Number of Predictions
Silhouette Coefficient is metric used to evaluate the quality of clustering.I It measures how similar a data point to its own cluster compared to other clusters, considering both cohesion and separa- tion. Values range from -1 to +1, where higher value indicates that the object is well-matched to its own cluster and poorly matched to neighboring clusters.
b(i) a(i)
sii)
(38)
max( b(i)
where:
a(1) is the average distance from sample all other points in the same cluster (intra-cluster dissimilarity).
b(t) is the minimum average distance from sample i to all points in any other cluster (inter-cluster dissimilarity).
Expert Overlap primarily describes a feature in Mixture of Experts (MoE) models where special- ized subnetworks (experts) are not entirely distinct. These experts might share parameters or have intentionally intersecting knowledge domains to process similar types of data or tasks.
The actual number of neighbors, K,, used for an input parameter kporam and N total embeddings is:
min min(kparama N 1)
(39)
The overlap score for an individual embedding ei, denoted O;, is:
O, =
Σ I(L, (1)
(40)
CGEN)(k)
The overall expert overlap score, Sonerlap- is the average of these individual scores:
Soverlop
0
Σ ICG I4)
(41)
N
K'
1=1
1=1
E,EN((K)
where:
N is the total number of embeddings.
kparam is the user-specified number of nearest neighbors.
k the adjusted number of nearest neighbors, min(kparams N - 1), used in the calculation (meaningful for k' 0).
e; represents the i-th embedding from the set of embeddings E {e1, ....,, eN
37

---

4 is the expert label corresponding to the embedding c, from the set of labels L
{4, 11,l2....,xx
(Kk)) is the set of k' nearest neighbors of embedding ex, excluding e; itself.
II() is the indicator function, which is 1 if the condition (e.g.. 4 # 4)) is true, and otherwise.
The Soverlap score ranges from 0 to A score of indicates overlap (all K' nearest neighbors of any point share its label), while a score of indicates complete overlap (all k' nearest neighbors of any point have different labels). A lower score generally signifies better expert separation the embedding space.
Routing Variance refers the inconsistency fluctuation in how the gating network distributes inputs to different expert sub-models. It measures the variability in which expert(s) are chosen for similar inputs over time, reflecting the stability of the routing decisions.
NE
Ns
RoutingV ariance
1
Σ
1
(42)
NE
Ns
NE
where:
Ng: Total number of experts.
Ns: Number of input samples.
Og (z:): Gating probability of input æ; being assigned to expert j.
Root Mean Square Error (RMSE) is standard statistical metric used evaluate the performance of a model by quantifying the magnitude of error between predicted and observed values. Lower RMSE values signify closer fit of the model the data, indicating higher predictive accuracy.
RMSE
U - 0.)²
(43)
where:
n is the total number of observations.
represents the i-th actual (observed) value.
haty; represents the i-th predicted value.
sum denotes the summation over all observations from
Implementation Details
Mfixture-of-Experrt Experts (MoE) language model with 16.4B parameters. It employs an innovative MoE architecture, which involves two principal strategies: ine-grained expert segmentation and shared experts isolation. is trained from scratch on 2T English and Chinese tokens, and exhibits comparable performance with DeekSeek 7B and LLaMA2 7B, 7B, with only about 40% of computations.
DeepSeek-Moe-160 DeepSeekMoE-166 is Mixture-of
Moonlight-16B-A3BB [88] Moonlig iigttightttoo 16B-A3B is a 16 billion-parameter Mixture-of ofe--xerts (MoE) language model developed by Moonshot AI. It employs the Muon optimizer train on 5.7 trillion tokens, achieving a new Pareto frontier of performance per FLOP. Available in both billion activated-pee parameter comparable models such as Llama. 3-3B and sesppseek-22-11 less compute. The model and its instruction-tumee variant open-source on Hugging Face, with checkpoints and a memory- and communication-effin further research.
activated-parrmeteer inference configuration and the full 16 billion-pane parameter scale, it outperforms
while requiring significantly
Muon implementation provided to foster
DeepSeek- [2] DeepSeek-v2 is a strong Mfxture-of-Exppets (MoF) language model characterized by economical training and efficient inference. DeepSeek-VZ adopts innovative
38

---

architectures including Multi- Latent Attention (MLA) and DeepSeekMoE. MLA guarantees effi- cient inference through significantly compressing the Key- Value (KV) cache into a latent vector, while DeepSeekMoF. enables training strong models at economical cost through sparse computation.
We integrate balance loss Lialance into each MoE. layer by modifying the model's modeling file. During training, due to device computational resource constraints, we employ LoRA for fine- tuning (note that the sole difference from full-parameter fine-tuning lies in the smaller number of parameters, with no fundamental difference in the training mechanism). LoRA uses standard configurations (rank 32, LoRA a 128, learning rate 10 batch size 32, dropout 0.1). We keep several original training hyperparameters in the model configuration, including the weight of Caux defaulting to 0.001. Settings like top-k activation and routing scoring function type match each model's s default configurations. The weights 3 and for our L. and C., are set identically to a. During inference, we first merge the trained LoRA into the base model, then infer using vllm with utilizzaiio
gop__mmory__o memory_utilization = 0.9. Evaluation uses three validation methods: rule-based extraction, GPT-4o, and human experts.
G Baselines
GShard [3] GShard is a pioneering Mixture-of-Exxpett (MoE) architecture developed by Google Research, designed for massively parallelized training across thousands of devices. It introduces automatic tensor sharding to scale model parameters and data efficiently, achieving dynamic load balancing during distributed computation. Trained on 600 billion tokens, GShard demonstrated break- through performance in multilingual machine translation across 100+ languages while maintaining linear computational cost scaling. Its innovations sparse expert routing and memory optimization laid the foundation for subsequent large-scale MoE systems.
ST-MoE[[55] ST-MoE (Sparsely-Traines Mixture-of-Experts Google that enhances MoE model stability through specialized training techniques. It employs a novel router design with expert dropout and auxiliary loss terms to prevent mode collapse during sparse activation. Scaling to 269 billion parameters with only billion active parameters per token, ST-MoE achieves state-of-the-ar results on language modeling and reasoning tasks while using 5-7x less compute than dense counterparts, The architecture incorporates parameter-sharing strategies across experts improve sample efficiency and reduce memory footprint.
compute-efficient framework from
Loss-Free Balancing[68 Loss-Free Balancing addresses the routing imbalance in MoE models without explicit optimization objectives. Traditional approaches rely on auxiliary loss functions to enforce expert load balancing, often at the cost of model performance or computational efficiency. This method dynamically adjusts entropy constraints on routing decisions and incorporates an adaptive activation threshold mechanism for sparse gating, achieving balanced expert utilization without auxiliary losses. It preserves primary task performance while demonstrating robustness in large-scale multi-task scenarios.
With Aux Loss[46] This classical load-balancing strategy for MoE training introduces explicit auxiliary losses during routing to constrain variance in expert utilization. Two complementary designs implemented: (1) soft regularization terms (e.g., L2 penalties) based on expert selection frequency, and (2) probability redistribution strategies for cold-start experts. While effective mitigating long-tail tail distribution issues, it requires careful tuning of loss weights avoid interference with the primary task.
H Experiments Details
H.1 Hyperparameter Sensitivity
To address the importance of hyperparameter sensitivity, we conducted experiments varying the values of the loss weights a (for Lauz). 3 (for Lo), and y (for L.) across different magnitudes.
For reference, our overall loss function L is defined as the sum of Ln, and Laatance- The balance loss Lbalance is defined as:
Ln + Libatace
(44)
Lhatance a Laux + B Lo + 7 L
(45)
39

---

It is worth noting that we apply a dynamic balancing mechanism to ensure fair weighting across different loss terms. Specifically, because the orthogonality and variance losses (L, and L,) may have different initial scales, we first normalize them using dynamic scaling factors. This brings their magnitudes roughly in line with the auxiliary loss Lauz- Only after this normalization do we apply the hyperparameters a, B, and y to control their contributions to the total loss.
The table below summarizes our results across several representative benchmarks under four different settings (DS v2 lite), as shown in Table 3.
Table 3: Hyperparameter sensitivity analysis. We evaluate performance across multiple benchmarks with different combinations of ax y (DS v2 lite).
a, 8,
MMLU
GPQA
HumanEval
GSM8K
MATH500
Max VioGlobba
10 10 10-3 10 35.59
28.76
43.58
50.94
49.33
2.52
10-3, 10 100-3
31.24
25.52
41.62
46.63
46.23
3.05
10-3.10-3.10 10-1 10 10-3,00031100
33.52
27.35
39.52
48.30
49.09
2.77
30.74
26.90
42.85
49.62
44.54
4.57
From the results in Table 3, we observe that the setting where - 3 = 10 the best performance across tasks. This suggests that the performance is optimal when all three loss weights a, B, and y set the same value.
consistently yields
Furthermore, our method demonstrates strong robustness across different hyperparameter magnitudes. When any of the coefficients is varied within order of magnitude (+1), i.c., 10 vs 10 the results remain stable and close to optimal. This indicates that our method is not overly sensitive to these hyperparameters and can be considered robust in practical applications.
H.2 Configurations and Base Model Performance
discrepancy between our reported results and the original model figures from public citations (e.g., Moonlight, DeepSeek) was observed. This disparity primarily arises from differences in model versions, prompting strategies, and inference settings. We clarify these differences below:
Model versions:
The public figures are typically based on instruction tuned models. In contrast, work starts from their pretrained base versions, which have preference or SFT (Supervised Fine-Tuning) data, leading to inherently different performance baselines.
Prompting strategies: Our evaluation is conducted in zero-shot setting without hand- crafted few-shot prompts or demonstrations, which are often used in official evaluations.
Inference length: We uniformly limit the generation to 512 max new tokens due to computational constraints. In contrast, official results often 8k-32k tokens, which notably benefits reasoning-heavy tasks like MMLU and HumanEval.
To quantify this impact, we evaluated both base and our fine-tuned models under the same, matched token budget (512 tokens). The results are summarized in Table We also analyze the effect of increasing the token length for the Kimi model in Table 5.
Table 4: Performance Comparison under Matched Inference Settings (512 max new tokens).
Method
MMLU
GPQA
HumanEval
GSM8K
MATH500
Max VioGloball
Base (ds)
28.46
22.45
42,64
28.76
7.34
4.63
Ours (ds)
33.35
25.15
63.30
35.00
10.82
2.19
Base (ds-v2)
26.56
20.33
31.34
22.57
15.69
6.97
Ours (ds-v2)
35.59
28.76
43.58
50.94
49.33
2.52
Base (Kimi)
34.23
28.33
55,67
81.23
53.76
8.37
Ours (Kimi)
40.36
32.01
70.64
87.62
59.64
7.23
40

---

Table 5: Effect of Increasing Max New Tokens (Kimi).
Method
MMLU
GPQA
HumanEval
GSM8K
MATH500
Max VioGlobal
Base 512
34.23
28.33
55.67
81.23
53.76
8.37
Base 1024
37.74
31,42
60.24
82.42
55.62
8.22
Base 2048
45.43
33.52
62.09
82.73
60.13
9.01
Ours 512
40.36
32.01
70.64
87.62
59.64
7.23
Ours 1024
45.63
35.22
73.23
88.31
65.23
7.14
Ours 2048
47.95
36.78
73.62
86.25
69.82
6.87
As shown in Table 4, under identical inference constraints (512 tokens), method consistently outperforms the original base models. Furthermore, Table 5 demonstrates that our method retains its leading performance even when the generation length is extended. We note that due computational resource constraints, we were unable to reproduce the official results from other papers, which often utilize significantly longer sequence lengths (e.g., 8k-32k tokens).
H.3 Performance Under Larger and More Diverse Training Data
We conducted an experiment to evaluate the impact of training data size and diversity on the effectiveness of method.
H.3.1 Motivation from Single-Tass Task Settings
As noted in the introduction, method is motivated by the observation that in post-training scenarios, the training data often Nomain-specinf and less diverse. This results highly skewed token distributions, which intensifies the conflict between load balancing (which encourages token-to expert allocation) and expert specialization (which encourages domain -specifc token routing). Our method designed to explicitly address this tension.
H.3.2 Performance on Mixed and Richer Datasets
To test whether our method still performs well with more diverse training data, we constructed mixed dataset combining Numina (math), GPQA (science), and HumanEval (coding), totaling 18k examples. We fine-tuned the Moonlight (Kimi) model for epochs on this combined dataset. The results are summarized in Table 6.
Table 6: Performance comparison larger, mixed dataset (Numina, GPQA, HumanEval) using the Moonlight (Kimi) model.
Method
MMLU
GPQA
HumanEval
GSM8K
MATH500
Max VixGlooba
Base
34.23
28.33
55.67
81.23
53.76
8.37
AuxOnly
36.98
31.34
67.53
84.83
62.29
7.07
AuxFree
35.87
29.48
68.83
86.29
63.84
7.28
Ours
45.38
37.01
78.93
92.92
67.83
7.11
As shown, our method continues to outperform all baselines, even when trained on a significantly larger and more diverse dataset. This demonstrates that approach remains robust and effective beyond constrained single-task settings.
I More Baselines and MoE Architectures
I.1 Comparison with Additional Baselines
To provide more comprehensive evaluation, expanded our set of comparison methods to include two additional of art baselines. We re-evaluated all methods on the most comprehensive subsets of our benchmark suite.
41

---

The added baselines are:
Dynamic Routing MoE (ERNIE 4.5) [5]: This is strong recent baseline that introduces a multimodal, heterogeneous MoE architecture. It supports both parameter sharing across modalities and -peeiit expert specialization. The ERNIE 4.5 family includes multiple model scales (e.g., 47B and 3B active parameters) and has shown competitive performance on various text and multimodal benchmarks.
SIMBAL (Similarity-Presse Preserving Routers) [55]: This is recent method addressing expert load balancing in sparse MoE models. Instead of enforcing uniform routing via conventional load balancing loss, SIMBAL introduces the router's s Gram matrix with the identity matrix, encouraging similar input tokens to be routed to similar experts, thereby reducing redundancy and improving consistency in expert utilization.
arthogonality-basee regularization. This aligns
A summary of the key results is presented in Table 7.
Table 7: Performance comparison against additional sate-offttheart baselines. Our method demon-
strates superior performance and achieves the best (lowest) load balance score (Max VioGlobal..
Method
MMLU
GPQA
HumanEval
GSM8K
MATH500
Max VioGlobal
ERNIE 4.5 LBL
32.44
27.45
37.32
47.24
42.63
3.45
SIMBAL
31.89
27.64
39.45
48.75
45.36
4.56
Ours
35.59
28.76
43.58
50.94
49.33
2.52
As shown in Table 7, after incorporating these two additional state-of-the-art baselines, our approach continues to deliver the best overall performance. Across six representative tasks, our method either matches surpasses the strongest new baseline, while simultancously maintaining the lowest Max VioGlobal (indicating better load balance). These additional results confirm that the improve- ments reported in the main paper are not artifact of the original bascline selection but hold against the latest alternatives as well.
1.2 Performance on Diverse MoE. Architectures
To further validate the generality of our method, we extended our evaluation to more diverse MoE architectures. Our initial experiments focused on DeepSeek and Moonlight models due to their strong open-source performance and recent community adoption. To broaden this scope, we additionally evaluated our method on two structurally different models: Mixtral and Phi-MoE, which adopt distinct routing strategies and omit shared experts.
The results, shown in Table 8, demonstrate that our method continues to outperform baselines across all tasks on these diverse architectures.
Table 8: Performance comparison diverse MoE architectures (Mixtral and Phi-MoE).
Method
MMLU GPQA
HumanEval
GSM8K
MATH500
Max VioGlobal
Mixtral Architecture
Mixtral-Bass
43.32
15.34
33,23
52.42
20.63
6.32
Mixtral-AuxOnly
50.56
18.84
39.74
58.73
28.74
3.25
Mixtral-AuxFree
49.73
20.14
36.06
56,96
29.84
3.58
Mixtral-Ours
52.63
20.74
37.73
61.74
33.42
3.54
Phi-MoE Architecture
PhiMoE-Base
51.73
34.52
66.46
84.52
41.84
7.53
PhiMoE-AuxOnly
57.24
34.21
71.32
85.21
42.94
5.21
PhiMoE-AuxFree
53.52
35.32
70.45
86.24
44.52
5.35
PhiMoE-Ours 59.63
35.87
76.23
88.32
44.79
5.32
These results further demonstrate the generality our approach, showing its effectiveness across MoE models with different underlying architectural designs.
42

---

Figure 5: Selected Images (4x3)
Training Overhead
While method introduces some additional computation due to the proposed regularization losses, the training time remains within a practical range and compares favorably with existing baselines. We report the average step time (in seconds per iteration) the DeepSeek V2 Lite model using batch size of 32. The results are summarized in Table 9.
Table Training time comparison (seconds per iteration) on the DeepSeek V2 Lite model (batch size 32).
Method
Time (s/iter)
Ours
11.5
Only Aux
10.7
Aux Free
9.8
GShard
14.8
ST-MoE
12.1
Our approach incurs moderate overhead compared boad-balancing-on methods like "Aux Free" and "Only Aux," but remains significantly more efficient than GShard and ST-MoE. Given that our method achieves up 23.79% performance improvement across benchmarks (as reported in the abstract), we believe this effiiiecyyperffrmmacce trade-off is well justified.
K
Visualization
Figures present the PCA projection of token embeddings assigned the top most active experts from baseline models. The significant overlap among different colors suggests that the token representations routed to different experts are well separated. This indicates high expert overlap and lack of clear specialization among experts in the representation space.
43


Reviews:
[{"review_id": "kk7di3pkXE", "reviewer": "Reviewer_jbyJ", "summary": "This paper addresses a critical challenge in Mixture-of-Experts (MoE) models: the trade-off between load balancing and expert specialization. The authors argue that the standard auxiliary load balancing loss often leads to overly uniform routing and expert overlap, which hinders model performance, especially during post-training adaptation. To tackle this issue, the paper proposes a novel method that introduces two complementary loss functions: an orthogonality loss to encourage experts to learn distinct representations for different tokens, and a variance loss to promote more discriminative routing decisions. The authors provide a theoretical analysis demonstrating the compatibility of these new losses with the existing auxiliary loss. Through extensive experiments on three different MoE models and eleven diverse benchmarks, the paper shows that the proposed method significantly enhances expert specialization and improves downstream task performance by up to 23.79% compared to baselines, without requiring any architectural modifications or compromising load balancing.", "questions": "1. On hyperparameter sensitivity: The choice of hyperparameters β and γ for the new loss functions seems crucial for the method's performance. Could you provide analysis on the sensitivity of your method to these hyperparameters?\n2. On compareing baselines: The paper compares against classical MoE methods (e.g., GShard, ST-MoE) and auxiliary loss-based balancing. Could you add more recent baseline results?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Novel and Impactful Contribution: The paper presents a simple yet effective solution to a well-recognized problem in MoE models. The proposed orthogonality and variance losses offer a principled way to directly encourage expert specialization, a crucial aspect for unlocking the full potential of MoE architectures. The potential for significant performance gains without architectural changes makes this work highly valuable for the community.\n\n- Comprehensive Experimental Evaluation: The empirical evaluation is a major strength of this paper. The authors have conducted experiments on a diverse set of MoE models (DeepSeek-Moe-16B, Moonlight-16B-A3B, and DeepSeek-V2-Lite) and a wide range of 11 benchmarks covering mathematics, multi-domain tasks, and code generation. The consistent improvements across different models and tasks provide strong evidence for the effectiveness and robustness of the proposed method. The inclusion of four relevant baselines further strengthens the experimental setup.\n\n- In-depth Analysis and Ablation Studies: The paper provides a good level of analysis, including a theoretical discussion on the compatibility of the proposed loss functions.\n\nWeaknesses:\n- Lack of Hyperparameter Sensitivity Analysis: The paper states that the hyperparameters for the new losses (β and γ) are set to the same value as the auxiliary loss weight (α=0.001). A sensitivity analysis or at least a discussion on the tuning process for these hyperparameters would significantly enhance the practical value of the proposed method.\n- Generalizability: The benchmarks are primarily language- and code-focused. Broader validation in multimodal or low-resource settings (hinted at in limitations) would strengthen the claims.", "was_revised": true, "final_justification": "The authors have made sincere efforts to address my concerns. However, I recommend maintaining my original (high) score because the revisions, while appreciated, do not significantly alter my initial assessment of the paper's strengths and contributions.", "rebuttal": "**Dear Reviewer jbyJ,**\n\nWe sincerely thank the reviewer for the thoughtful evaluation and detailed feedback. We are especially grateful for the recognition of our paper’s strengths, including the novelty and impact of our method, the comprehensive empirical evaluation across multiple models and tasks, and the depth of our theoretical and ablation analyses.\n\nAt the same time, we also appreciate the reviewer’s constructive suggestions. The main concerns can be summarized as follows:\n\n1. The lack of hyperparameter sensitivity analysis (experimental);\n2. The task scope is limited to NLP, and broader validation in multimodal or low-resource settings is needed (discussion);\n3. More recent baselines should be included for comparison (experimental).\n\nWe have carefully addressed each of these points in the following responses.\n\n---\n\n**1. On hyperparameter sensitivity**\n\n**Conclusion:**\n The performance is optimal when the loss weights $\\alpha$, $\\beta$, and $\\gamma$ are set to the same value. Moreover, our method demonstrates strong robustness across different hyperparameter magnitudes.\n\n**Explanation:**\n\nWe thank the reviewer for highlighting the importance of hyperparameter sensitivity. To address this, we conducted experiments varying the values of $\\alpha$ (for $\\mathcal{L}_{aux}$), $\\beta$ (for $\\mathcal{L}_o$), and $\\gamma$ (for $\\mathcal{L}_v$) across different magnitudes.\n\nFor reference, our overall loss function is defined as:\n\n$\\mathcal{L}=\\mathcal{L} _ h +\\mathcal{L}_{balance}$\n\n$\\mathcal{L} _ {balance} = \\alpha \\cdot \\mathcal{L}_{aux} + \\beta \\cdot \\mathcal{L} _ {o} + \\gamma \\cdot \\mathcal{L} _ {v}$\n\n\nIt is worth noting that we apply a *dynamic balancing mechanism* to ensure fair weighting across different loss terms. Specifically, because the orthogonality and variance  losses ( $\\mathcal{L} _ {o}$ and $\\mathcal{L} _ {v}$ ) may have different initial scales, we first normalize them using dynamic scaling factors. This brings their magnitudes roughly in line with the auxiliary loss $\\mathcal{L}_{aux}$. Only after this normalization do we apply the hyperparameters $\\alpha$, $\\beta$, and $\\gamma$ to control their contributions to the total loss.\n\nThe table below summarizes our results across several representative benchmarks under four different settings (DS v2 lite):\n\n| $\\alpha, \\beta, \\gamma$ | MMLU  | GPQA  | HumanEval | GSM8K | MATH500 | MaxVioGlobal |\n| ----------------------- | ----- | ----- | --------- | ----- | ------- | ------------ |\n| 1e-3, 1e-3, 1e-3        | 35.59 | 28.76 | 43.58     | 50.94 | 49.33   | 2.52         |\n| 1e-3, 1e-4, 1e-3        | 31.24 | 25.52 | 41.62     | 46.63 | 46.23   | 3.05         |\n| 1e-3, 1e-3, 1e-4        | 33.52 | 27.35 | 39.52     | 48.30 | 49.09   | 2.77         |\n| 1e-4, 1e-3, 1e-3        | 30.74 | 26.90 | 42.85     | 49.62 | 44.54   | 4.57         |\n\nWe observe that the setting where $\\alpha = \\beta = \\gamma = 10^{-3}$ consistently yields the best performance across tasks. Additionally, when any of the coefficients is varied within one order of magnitude ($\\pm 1$), the results remain stable and close to optimal. This indicates that our method is not overly sensitive to these hyperparameters and can be considered robust in practical applications.\n\n---\n\n**2. On Generalizability**\n\n**Conclusion:**\n\nWe believe our formulation is inherently suitable for both multimodal and low-resource scenarios, as its design principles, such as expert specialization and routing diversity, are broadly applicable across domains.\n\n**Explanation:**\n\nWe appreciate the reviewer’s insightful suggestion regarding broader evaluation. While our current experimental design focuses on NLP and code generation tasks, we believe our proposed training objective is agnostic to modality and can be extended to other domains, such as visual and multimodal models. The orthogonality and variance principles are applicable wherever a mixture-of-experts architecture is employed, including vision MoEs and multimodal encoders/decoders.\n\n- **Visual / multimodal MoEs.**\n  Recent works [1, 2, 3] on vision‑centric experts shows that image patches suffer from the *same* expert‑overlap and uniform‑routing issues we address in language models. Because our orthogonality loss acts directly on expert outputs and our variance loss on routing logits, both can be applied to visual tokens without modification. \n\n- **Low‑resource adaptation.**\n  Parameter‑efficient fine‑tuning methods such as MoELoRA [4], MixLoRA [5], and MoKA [6] insert small trainable adapters while freezing most expert weights. Our losses require *no* additional parameters and can be computed on the adapted weights or the LoRA pathways alike, making them a natural drop‑in to further improve specialization when GPU memory or data are limited.\n\n---\n\n**3. On comparing baselines**\n\n**Conclusion:**\n After incorporating two additional state‑of‑the‑art baselines, our approach continues to deliver the best overall performance.\n\n**Explanation:**  \nWe appreciate the reviewer’s suggestion to expand the set of comparison methods. In response, we have added:\n\n- **Dynamic Routing MoE (ERNIE 4.5) [7]** is a strong recent baseline. It introduces a multimodal, heterogeneous MoE architecture that supports both parameter sharing across modalities and modality-specific expert specialization. The ERNIE 4.5 family includes multiple model scales (e.g., 47B and 3B active parameters), and has shown competitive performance on various text and multimodal benchmarks.\n\n- **SIMBAL (Similarity-Preserving Routers) [8]** is a recent method that addresses expert load balancing in sparse MoE models. Instead of enforcing uniform routing via conventional load balancing loss, SIMBAL introduces an orthogonality-based regularization that aligns the router’s Gram matrix with the identity matrix. This encourages similar input tokens to be routed to similar experts, reducing redundancy and improving consistency in expert utilization.\n\n\nWe re‑evaluated all methods on the most comprehensive subsets of our benchmark suite. A summary of the key results is presented below.\n\n| Method                             | MMLU  | GPQA  | HumanEval | GSM8K | Math500 | MaxvioGlobal |\n| ---------------------------------- | ----- | ----- | --------- | ----- | ------- | ------------ |\n| ERNIE 4.5 LBL | 32.44 | 27.45 | 37.32     | 47.24 | 42.63   | 3.45         |\n| SIMBAL                 | 31.89 | 27.64 | 39.45     | 48.75 | 45.36   | 4.56         |\n| **Ours**                           | 35.59 | 28.76 | 43.58     | 50.94 | 49.33   | 2.52         |\n\n*(Numbers will be filled in Table 1 of the revision.)*\n\nAcross all six representative tasks, our method either matches or surpasses the strongest new baseline, while simultaneously maintaining the lowest MaxVioGlobal (better load balance). These additional results confirm that the improvements reported in the main paper are not an artifact of the original baseline selection but hold against the latest alternatives as well.\n\n\n---\n\nWe are grateful for the reviewer's insightful feedback, which has been instrumental in strengthening our work. We have strived to thoroughly address every concern with new experiments and detailed clarifications, and we are looking forward to the opportunity for any further discussion.\n\n---\n\n[1] Mustafa, B., Riquelme, C., Puigcerver, J., Jenatton, R., & Houlsby, N. (2022). Multimodal Contrastive Learning with LIMoE: The Language‑Image Mixture of Experts. Advances in Neural Information Processing Systems, 35, 9564–9576.\n\n[2] Yan, Z., Li, Z., He, Y., Wang, C., Li, K., Li, X., Zeng, X., Wang, Z., Wang, Y., & Qiao, Y. (2025). Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment. Proceedings of the Computer Vision and Pattern Recognition Conference, 29880–29892.\n\n[3] Li, Z., Li, Z., & Zhou, T. (2025). R2‑T2: Re‑Routing in Test‑Time for Multimodal Mixture‑of‑Experts. arXiv.\n\n[4] Luo, T., Lei, J., Lei, F., Liu, W., He, S., Zhao, J., & Liu, K. (2024). MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter‑Efficient Fine‑Tuning for Large Language Models. arXiv.\n\n[5] Li, D., Ma, Y., Wang, N., Ye, Z., Cheng, Z., Tang, Y., Zhang, Y., Duan, L., Zuo, J., & Yang, C. (2024). MixLoRA: Enhancing Large Language Models Fine‑Tuning with LoRA‑Based Mixture of Experts. arXiv.\n\n[6] Yu, B., Yang, Z., & Yi, X. (2025). MoKA: Parameter Efficiency Fine‑Tuning via Mixture of Kronecker Product Adaption. Proceedings of the 31st International Conference on Computational Linguistics, 10172–10182.\n\n[7] ERNIE Team (Baidu). (2025, June 29). ERNIE 4.5 Technical Report. Retrieved from Huggingface.\n\n[8] Omi, N., Sen, S., & Farhadi, A. (2025, June 16). Load Balancing Mixture of Experts with Similarity Preserving Routers. arXiv."}, {"review_id": "kFawVuJu1v", "reviewer": "Reviewer_9cq6", "summary": "This paper makes an observation that experts often lack specialization (e.g., the input gets routed to experts uniformly) due to load balancing, and introduces two auxiliary loss terms that encourage specialization without sacrificing load balancing – (1) an orthogonality loss, which reduces the similarity between experts that receive the same token, and (2) a diversity loss, which discourages expert weights from being uniformly distributed. Experiments on 11 benchmarks, four baseline training methods, and four base models show that the proposed model consistently and substantially outperforms all baselines, while maintaining effective load balancing.", "questions": "(Questions are in the Strengths And Weaknesses section)", "limitations": "(Limitations are in the Strengths And Weaknesses section)", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Overall, I found this paper to be very well written and strongly motivated. The subtle tension between expert specialization and load balancing is an important and underexplored issue, and this paper does an excellent job articulating it with clear and convincing explanations. While MoEs are widely studied in general, this problem of encouraging expert specialization is underexplored, and remaining having balanced load is very critical for practical reasons, e.g., training stability. Addressing both together is very impressive.\n- The paper also clearly explains the key observations and the technical proposals that follow from them. Admittedly, I did not carefully check every math detail line by line, but I was able to understand the underlying motivation and that the math equation aligns with that motivation.\n- Experiments are very thorough and comprehensive, quite beyond typical standards for a conference paper. Evaluating on eleven benchmarks, four baselines, and three base models, with consistent and substantial improvements across the board, is very impressive. The detailed analysis is also very valuable.\n\nWeaknesses\n- My most critical concern — despite all the strengths noted above — is that the actual training setup and data used are very unclear. First of all, it only appears in the *appendix* that the paper doesn’t actually do full training but it is doing LORA fine-tuning. This came as a surprise, as the rest of the paper is written as full training was performed. This is very different from taking the existing base models and applying LoRA tuning. This should be clearly stated throughout the paper — including the abstract, introduction, methodology, and experiments. And, this leads to a follow up question: how much (a small number of) additional training steps with LoRA fundamentally changes how experts are trained, as expert weights themselves won’t be updated?\n- In fact, even the fact that the paper is mainly targeting post-training was unclear, e.g., the abstract and the intro mentions the word “post-training” once each without much contextualization, so if readers miss this word, the entire paper is written as it is pre-training. Actually, is there a reason there’s a fundamental difference between post-training and pre-training, so that expert specialization is more important in post-training than pre-training, given that pre-training and post-training are actually the same training process except the data and where the loss is applied to are different?\n- Aside from these, other training setups are unclear either. Is the model trained in a multi-task setup using a single training dataset across all elevent benchmarks, or is there a separate training dataset and a separate trained model for each benchmark? How many training steps are used? What is the performance of the base model before LoRA fine-tuning (e.g., in Table 1)? These are important information that impact the implication of the methods and experimental results drastically, that are currently missing.\n\nMinor weaknesses/questions\n- Orthogonality loss: just to confirm my understanding – this loss encourages the experts that receive the same token to be less similar to each other, so that a single token benefits from greater diversity? If that’s correct, it feels more about “expert diversification” rather than “expert specialization”. To me, specialization implies that an expert becomes responsible for (one or more) domains or token types.\n- I think Figure 1 can significantly be improved - it is quite complex and is not intuitive enough.", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer 9cq6,**\n\nWe sincerely thank the reviewer for the thoughtful and encouraging feedback. We appreciate the recognition of our motivation, the clarity of our formulation, and the importance of jointly addressing expert specialization and load balancing. Below we summarize our responses to the main concerns:\n\n1. The actual use of LoRA-based fine-tuning instead of full training is unclear and should be stated explicitly;\n2. The distinction between pre-training and post-training lacks contextual clarity and justification;\n3. Key experimental details such as data splits, training steps, and base model performance are missing;\n4. The explanation of the orthogonality loss may conflate diversification and specialization;\n5. Figure 1 is overly complex and not intuitive.\n\nWe have carefully addressed each of these points in the following responses.\n\n---\n\n**1. On the Use of LoRA Fine-Tuning**\n\n**Conclusion:**\n\nAlthough LoRA is used for parameter-efficient training, all major components (including experts and router) are updated during fine-tuning, making our setup functionally equivalent to full-model training.\n\n**Explanation:**\nWe thank the reviewer for this important and constructive comment. We acknowledge that the current version may be misleading, as the use of LoRA was only stated in the appendix. In the revision, we will clearly indicate in the abstract, introduction, methodology, and experiments that all training is conducted via LoRA-based fine-tuning.\n\nIn our setup, LoRA modules are inserted into **both the router and the expert layers**. This means that the routing behavior and expert outputs are both optimized during training. While LoRA offers parameter-efficient adaptation, it does not freeze the underlying model. On the contrary, all core components are updated, just through a low-rank pathway. To reduce the gap between LoRA and full fine-tuning, we have set the LoRA rank to 32, which is relatively high and helps better approximate full updates.\n\nTherefore, our training is functionally equivalent to full fine-tuning in terms of which parameters are optimized. Importantly, our proposed losses directly act on trainable parts of the model: the variance loss shapes the router logits, and the orthogonality loss regularizes expert outputs. The observed improvements in all benchmarks validate their effectiveness under this setup *(The specific training details are discussed in detail in Response 4)*.\n\nWe appreciate the reviewer’s attention to this detail and will make the training design and parameter update scope explicit in the revised paper.\n\n---\n\n**2. On the Clarity of the Post-Training Setting**\n\n**Conclusion:**\n\nTheoretically, our method is applicable to both pre-training and post-training phases. However, we focus on the post-training stage, where the conflict between expert specialization and routing balance becomes more pronounced in downstream tasks.\n\n**Explanation:**\n\nWe appreciate the reviewer’s insightful comment regarding the training stage targeted by our work. We would like to clarify the rationale for focusing on the post-training setting:\n\n- **On the distinction:** While we agree that pre-training and post-training share similar optimization processes, their goals differ. Pre-training aims to build general-purpose capabilities on diverse and large-scale corpora, whereas post-training seeks to adapt the model to specific domains or tasks. The data in post-training is often **less diverse and more skewed** in distribution compared to the **more uniform and balanced** nature of pre-training data.\n- **On the conflict:** In such downstream scenarios, token distributions can be highly concentrated (e.g., math tasks contain a higher proportion of numeric or symbolic tokens). This makes the conflict between load balancing and expert specialization more severe: load balancing encourages tokens to be evenly distributed across experts, while specialization requires assigning certain types of tokens more often to specific experts.\n- **Our motivation:** Our work is motivated by this core conflict. By introducing the orthogonality loss ($\\mathcal{L}_o$) and the variance loss ($\\mathcal{L}_v$), we aim to achieve better expert specialization without sacrificing load balance. This enables MoE models to better adapt to downstream distributions and is supported by both our theoretical insights and empirical results.\n\nIn the revised version, we will clearly indicate our focus on the post-training setting in the abstract and introduction, and explain why resolving this trade-off is particularly critical in this phase.\n\n---\n\n**3. On Training Setup, Task Scope, and Base Model Performance**\n\n**Conclusion:**\nEach benchmark is trained separately using a high-quality 6k-sample dataset. Fine-tuning is limited to 3 epochs (~550 steps) per task. Base model performance before fine-tuning will be added to Table 1 for clarity.\n\n**Explanation:**\n\n- **Data and task setup:** For each benchmark, we constructed a high-quality training set of 6,000 examples. The questions were sourced from the training split of the corresponding benchmark; when insufficient, we supplemented them with related datasets. Answers were generated using strong models such as OpenAI’s o3mini and DeepSeek R1, and then manually filtered to ensure correctness. Each dataset is used to fine-tune the model specifically for that benchmark.\n- **Training steps:** To avoid overfitting, we limited fine-tuning to 3 epochs across all tasks. Given the dataset size, this corresponds to approximately 550 training steps per model per task.\n- **Base model performance:** We have now included the base model performance before fine-tuning in Table below, using consistent settings across models (e.g., max sequence length = 512). This comparison shows that our proposed losses lead to consistent and significant improvements across all tasks, beyond the effect of LoRA tuning alone.\n\n| method | MMLU | GPQA | HumanEval | GSM8K | Math500 | MaxvioGlobal |\n| --- | --- | --- | --- | --- | --- | --- |\n| Base model DS | 28.46 | 22.45 | 42.64 | 28.76 | 7.34 | 4.63 |\n| ours on DS | 33.35 | 25.15 | 63.30 | 35.00 | 10.82 | 2.19 |\n| Base model DS_v2 | 26.56 | 20.33 | 31.34 | 22.57 | 15.69 | 6.97 |\n| ours on DS_v2 | 35.59 | 28.76 | 43.58 | 50.94 | 49.33 | 2.52 |\n| Base model Kimi | 34.23 | 28.33 | 55.67 | 81.23 | 53.76 | 8.37 |\n| ours on Kimi | 40.36 | 32.01 | 70.64 | 87.62 | 59.64 | 7.23 |\n\nWe appreciate the reviewer’s attention to these important experimental factors and will ensure that all relevant details are properly documented in the revised version.\n\n---\n\n**4. On Whether Orthogonality Loss Encourages Diversification or Specialization**\n\n**Conclusion:**\n Orthogonality loss directly promotes **expert diversification** at the token level, which serves as a critical mechanism for achieving **specialization** at the dataset level.\n\n**Explanation:**\n We appreciate the reviewer’s insightful interpretation. The orthogonality loss ($\\mathcal{L}_o$) encourages experts that receive the same token to produce diverse outputs. This diversity is not the final objective, but a necessary step toward achieving expert specialization.\n\n- **From diversification to specialization:**\n  1. When the same token is routed to multiple experts, $\\mathcal{L}_o$ penalizes similarity in their output representations, effectively encouraging the experts to respond differently. This is what we refer to as **expert diversification**.\n  2. When such pressure is applied across all tokens during training, experts are discouraged from learning redundant or overlapping representations. Instead, each expert is incentivized to capture distinct features or patterns in the data.\n  3. Over time, this token-level diversification aggregates into **specialization** across the dataset: each expert becomes better at handling certain types of inputs or semantic regions, resulting in stable expert-role assignment.\n- **Empirical evidence:**\n   This process is empirically supported in our experiments. As shown in Figure 1 (right), the token distribution across experts becomes significantly more clustered after training, with **reduced overlap** in representation space. Quantitatively, our method lowers expert overlap by up to 45%, indicating a strong shift toward specialization.\n\nIn short, diversification is the driving force, and specialization is the outcome. We appreciate the reviewer for raising this important distinction.\n\n---\n\n**5. On Figure 1 Readability and Design**\n\n**Conclusion:**\n We agree that the current version of Figure 1 may be visually dense and not sufficiently intuitive. In the revised version, we will redesign the layout, simplify visual elements, and add clearer annotations to improve overall readability. Below, we briefly explain the intended structure and purpose of the current figure.\n\n**Explanation:**\n Figure 1 is designed to visualize the two core effects of our method:\n\n- **Left side: Routing Diversification**\n  - *Top (Routing Score Matrix):* Shows the raw routing scores, which may appear abstract without further guidance.\n  - *Middle (Load Variance):* Demonstrates that our method preserves load balancing, achieving similar variance reduction as baseline methods.\n  - *Bottom (Discriminative Routing Variance):* This is the key part—our method significantly increases routing output variance, indicating more discriminative and less ambiguous routing decisions.\n- **Right side: Expert Specialization**\n  - *Bottom (Overlap):* Visualizes the heavy overlap of token assignments across experts in baseline methods.\n  - *Top (Cluster Separation):* In contrast, our method produces clearer token clusters per expert, indicating effective specialization and division of responsibility.\n\n---\n\nOnce again, we thank the reviewer for the thoughtful and constructive feedback. We believe our detailed responses and clarifications have thoroughly addressed the concerns raised. We look forward to any further discussion."}, {"review_id": "mVLHfAn8Pt", "reviewer": "Reviewer_4E1Q", "summary": "This paper investigates the issues of uniform expert routing and insufficient expert specialization in MoE models during the post-training stage, primarily caused by the auxiliary load-balancing loss. To address this, the authors propose two additional regularization objectives that respectively promote expert diversity and enhance router confidence. Experimental results demonstrate that the proposed approach improves model performance on downstream tasks.", "questions": "See Weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength:\n1. The idea of jointly optimizing expert specialization and routing distribution via additional regularization terms is interesting.\n\n2. The paper is well written, with clear motivations and a comprehensive discussion of the identified issues.\n\n3. The effectiveness of the proposed method is supported by empirical results.\n\n### Weakness:\n\n1. While Lemma 1 and Lemma 2 provide theoretical guarantees for the existence of a desirable routing configuration, their practical realization strongly relies on sufficient diversity in the input tokens. The paper lacks discussion on whether this assumption holds in real-world scenarios or how it may affect performance.\n\n2. The orthogonality loss \\( \\mathcal{L}_o \\), while effective in promoting expert specialization, introduces significant computational overhead due to its quadratic complexity with respect to the number of experts. Specifically, it requires computing pairwise projections between all top-\\(k\\) expert outputs for each input token, resulting in an overall cost of \\( O(N \\cdot k^2 \\cdot d) \\), where \\( N \\) is the token count, \\( k \\) the number of selected experts, and \\( d \\) the hidden size. This can become a bottleneck in large-scale MoE settings with many experts or long sequences. \n\n3. Currently, the proposed regularization losses are applied only during post-training. However, prior work such as OLMoE suggests that auxiliary losses may not be necessary during post-training at all. It would be valuable to discuss whether similar regularization should be incorporated during pretraining to promote expert specialization and more confident routing from the outset.", "was_revised": true, "final_justification": "Based on the overall evaluation of the paper, I have decided to maintain my initial positive score.", "rebuttal": "**Dear Reviewer 4E1Q,**\n\nWe sincerely thank the reviewer for their positive and constructive feedback. We are glad that the reviewer found our idea of jointly optimizing expert specialization and routing distribution via auxiliary regularization losses to be interesting. At the same time, we appreciate the reviewer’s thoughtful suggestions. The main concerns can be summarized as follows:\n\n  1. The effectiveness of the proposed loss may rely on token-level diversity, which may not always hold in real-world scenarios;\n  2. The orthogonality loss introduces non-negligible computational overhead, which may become a bottleneck in large-scale settings;\n  3. Prior work suggests that auxiliary losses may not be necessary during post-training, it is unclear whether regularization is more appropriate during pre-training.\n\n---\n\n**1. On the Assumption of Token Diversity**\n\n**Conclusion:**\n  Token diversity is a helpful condition, but not a necessary prerequisite for the success of our method. Through orthogonality loss ($\\mathcal{L}_o$) and variance loss ($\\mathcal{L}_v$), our approach actively induces functional diversity among experts, thereby enabling specialization even when input tokens are not highly diverse.\n\n**Explanation:**\n  We appreciate the reviewer’s insightful question regarding the connection between token diversity and the effectiveness of our method. Our responses are as follows:\n\n- **Active induction vs. passive dependence:**\n  Rather than relying on input diversity, our method actively creates internal diversity. The orthogonality loss ($\\mathcal{L}_o$) operates at the token level, penalizing similar outputs from different experts even when the same token is routed to them. This drives the experts to learn distinct representations, thereby fostering **functional diversity** even under low input diversity.\n- **Theoretical soundness and empirical robustness:**\n  *Lemma 1* and *Lemma 2* serve to establish that our **multiple training objectives** (load balancing and routing diversity) are theoretically compatible and jointly optimizable. Our loss design realizes this in practice. Most importantly, across 11 real-world benchmarks, including both high and low-diversity tasks, our method achieves consistent improvements, demonstrating strong robustness.\n\nWe hope this clarifies how our approach remains effective even without assuming ideal token diversity.\n\n---\n\n**2. On the Computational Overhead of $\\mathcal{L}_o$**\n\n**Conclusion:**\n  While $\\mathcal{L}_o$ has quadratic complexity in theory, the actual overhead is negligible in practice due to the small number of activated experts ($k$) and efficient batched implementations. It does not present a bottleneck in our setup.\n\n**Explanation:**\n  We appreciate the reviewer’s precise analysis of the computational cost of $\\mathcal{L}_o$, which involves pairwise projections among the $k$ selected expert outputs for each token, resulting in a theoretical cost of $O(N \\cdot k^2 \\cdot d)$. In practice, this cost remains manageable for several reasons:\n\n- **Small $k$ in standard MoE practice:**\n  One of the core design principles of sparse MoE models is to keep $k$ small to control computation. In our experiments, we follow this convention, using configurations such as $k=6$ in DeepSeek-V2-Lite. Given that $k \\ll N$ and $k \\ll d$, the quadratic factor in $k$ contributes minimally to total training cost.\n- **Efficient hardware execution:**\n  The main operations required by $\\mathcal{L}_o$, inner products and pairwise projections, are highly parallelizable and can be implemented as batched matrix multiplications. These are well-optimized in modern frameworks like PyTorch and run efficiently on contemporary GPUs.\n- **Justified by empirical gains:**\n  Most importantly, the modest increase in computation is offset by **substantial and consistent performance improvements** across diverse downstream tasks (e.g., Table 1 and Figure 4). This demonstrates that the regularization effect of $\\mathcal{L}_o$ leads to meaningful gains in expert specialization.\n\nWe will include a detailed breakdown of the computational cost of $\\mathcal{L}_o$ under typical $k$ values in the appendix of the revised version, to further clarify its practical feasibility.\n\n---\n\n**3. On the Applicability During Pretraining**\n\n**Conclusion:**\n\nWhile our current focus is on post-training, where the tension between uniform routing and specialization is most pronounced, we believe applying our regularization objectives during pretraining is a valuable future direction.\n\n**Explanation:**\n  We appreciate the reviewer’s insightful comparison with works such as OLMoE and the thoughtful reflection on different training phases.\n\n- **Why focus on post-training:** In the post-training stage, downstream tasks often involve narrow or skewed data distributions. This intensifies the conflict between load balancing (which pushes toward uniform token distribution) and the need for expert specialization. Therefore, it provides a clearer setting to observe and evaluate the effects of our proposed losses.\n- **Relation to “no auxiliary loss” claims:** Our work does not contradict the finding that traditional $\\mathcal{L} _ {aux}$ may be ineffective—or even detrimental—during post-training. As shown in our ablation results (Figure 4), the “w/o all” variant (which removes $\\mathcal{L} _ {aux}$) actually outperforms the “only aux” baseline. This confirms that $\\mathcal{L} _ {aux}$ may hinder specialization in post-training. Importantly, we further demonstrate that our proposed $\\mathcal{L} _ o$ and $\\mathcal{L} _ v$ actively promote specialization beyond simply removing constraints, achieving the best performance overall.\n- **Pretraining potential:** Our regularization framework is theoretically applicable during pretraining as well. In fact, we believe that applying $\\mathcal{L} _ o$ and $\\mathcal{L} _ v$ from the start may help guide expert specialization more effectively. We will highlight this in Section 6 (Limitations & Future Directions) as an important avenue for future exploration.\n\n---\n\nWe thank the reviewer again for the insightful and constructive feedback. We believe our detailed responses have thoroughly addressed each of the points raised. We look forward to any further discussion."}, {"review_id": "tyewMteMjq", "reviewer": "Reviewer_Ypny", "summary": "In this paper, the authors address the challenge of poor expert specialization in standard MoE models trained with load-balancing loss. The authors argue that the conventional load-balancing auxiliary loss leads to expert overlap or uniform routing. The paper proposes to mitigate that using two additional loss functions: 1) an orthogonality loss to encourage experts to learn unique functions and 2) a variance loss to promote more discriminative routing. The newly added loss functions are used for finetuning three pre-existing MoE models without any architecture modifications. Results are presented for 11 benchmarks including language, reasoning, science, math, and code evaluation.", "questions": "See the section above.\n\nMinor point:\nLine 66: Two periods at the end of the sentence.", "limitations": "The authors do not sufficiently address the limitations of their work, including discussion of the computational overhead for the added loss functions and a sensitivity analysis for MoE training with 4 loss functions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- The paper is well-written and addresses a significant, well-known conflict in MoE training.\n- The results show the model outperforms alternative regularization methods for the finetuned models.\n- The ablation showing the impact of different loss functions on the behavioural states of the models is very informative.\n\n**Weaknesses:**\n- **Missing pre-finetuning baseline results**: The authors do not report the performance of the original, pre-finetuning MoE models. Based on public figures for models like Moonlight and DeepSeek, the finetuning process described in the paper appears to significantly degrade performance on key benchmarks (e.g., MMLU, BBH). For example, the original accuracy of Moonlight MoE model on MMLU, MMLU pro, MBPP, and BBH are 70, 42.4, 63.8, 65.2, which are above the best reported numbers in Table 1. Similar behaviour is observed for DeepSeek models: MMLU 58.3 compared to the 35.59 of the proposed method, or BBH 44.1 compared to 38.84. \n\n- **LoRA mitigating performance loss**: The authors focus exclusively on LoRA finetuning of high-quality, pre-converged models, but this crucial detail is confined to the appendix. Given the high performance of the original base models, it should be clarified whether their method is simply mitigating performance loss during finetuning rather than providing a net gain over the original models.\n\n- **Complexity of multi-objective training with 4 loss functions:** While the authors only focus on finetuning already converged high-quality MoE models, it is unclear how the complex optimization of four separate loss functions behaves when training MoEs from scratch or when upcycling dense models, which are more challenging and common scenarios. The paper also lacks a sensitivity analysis for the loss coefficients (`α, β, γ`), making reproducibility and application difficult.\n\n- **Computational Overhead:** The paper does not discuss the computational overhead of the proposed loss functions. In particular, the orthogonality loss, which requires pairwise comparisons of expert outputs for the top-k selections, appears prohibitively expensive, especially with the top-6 expert configuration used in the experiments. A discussion of the added training cost and memory is essential.\n\n- **Limited MoE architectures:** All three models evaluated (DeepSeek, DeepSeek-Lite, Moonlight) share an identical MoE structure (64 experts, top-6 selection, 2 shared experts). To strengthen their claims, the authors should test their method on more diverse architectures like Mixtral and Phi-MoE, which use different routing strategies and do not include shared experts.", "was_revised": true, "final_justification": "The authors have addressed several of my concerns, including adding information on training settings, reporting overhead, and reporting downstream task performance.", "rebuttal": "**Dear Reviewer Ypny,**\n\nWe sincerely thank the reviewer for the thoughtful and detailed feedback. The comments raise several key issues, including: \n\n1. The lack of pre-finetuning baselines and potential performance drop; \n\n2. Whether the method mitigates LoRA-induced degradation; \n\n3. The complexity and reproducibility of multi-loss optimization; \n\n4. Missing analysis of computational overhead; \n\n5. Limited architecture diversity; \n\n6. Insufficient discussion of limitations.\n\nWe appreciate these valuable insights and address each point below.\n\n---\n\n**1. Missing pre-finetuning baseline results and apparent performance drop**\n\nThe discrepancy between our reported results and the original model figures primarily arises from differences in model versions, prompting strategies, and inference settings.\n\n- **Model versions**: The public figures cited (e.g., Moonlight, DeepSeek) are based on *instruction-tuned* models, while we start from their *pretrained base versions* with no preference or SFT data, leading to inherently different performance baselines.\n- **Prompting strategies**: Our evaluation is conducted in a **zero-shot** setting without handcrafted few-shot prompts or demonstrations, which are often used in official evaluations.\n- **Inference length**: We uniformly limit the generation to **512 max new tokens** due to computational constraints. In contrast, official results use **8k–32k** tokens, which notably benefits reasoning-heavy tasks like MMLU and HumanEval. To quantify this impact, we evaluated both base and our fine-tuned models under the same token budget. Results are summarized below:\n\n*Performance Comparison under Matched Inference Settings*\n\n|Method     |MMLU |GPQA |HumanEval|GSM8K|Math500|MaxvioGlobal|\n|---|---|---|---|---|---|---|\n|Base (ds) |28.46|22.45|42.64|28.76|7.34 |4.63|\n|Ours (ds) |33.35|25.15|63.30|35.00|10.82|2.19|\n|Base (ds-v2) |26.56|20.33|31.34|22.57|15.69|6.97|\n|Ours (ds-v2) |35.59|28.76|43.58|50.94|49.33|2.52|\n|Base (Kimi) |34.23|28.33|55.67|81.23|53.76|8.37|\n|Ours (Kimi) |40.36|32.01|70.64|87.62|59.64|7.23|\n\n\n\n*Effect of Increasing Max New Tokens (Kimi)*\n\n|Method    |MMLU |GPQA |HumanEval|GSM8K|Math500|MaxvioGlobal|\n|---|---|---|---|---|---|---|\n|Base 512 |34.23|28.33|55.67|81.23|53.76|8.37|\n|Base 1024|37.74|31.42|60.24|82.42|55.62|8.22|\n|Base 2048|45.43|33.52|62.09|82.73|60.13|9.01|\n|Ours 512 |40.36|32.01|70.64|87.62|59.64|7.23|\n|Ours 1024|45.63|35.22|73.23|88.31|65.23|7.14|\n|Ours 2048|47.95|36.78|73.62|86.25|69.82|6.87|\n\nAs shown, under identical inference constraints, our method outperforms the original models, and retains leading performance even when length is extended. We will include these clarifications and additional results in the revised manuscript. Due to computational resource constraints, we were unable to reproduce the official results with longer sequence lengths, and we sincerely apologize for this limitation.\n\n---\n\n**2. Clarifying Performance Gains with Our Method**\n\nWe would like to clarify that our approach provides **a net performance gain**, not just recovery from potential LoRA-induced loss. \n\nFirst, from a theoretical perspective, the proposed losses $\\mathcal{L}_o$ and $\\mathcal{L}_v$ are designed to actively enhance expert specialization while preserving load balance (as supported by Lemmas 1 and 2), which defines a constructive optimization objective beyond damage control (Detail in Reviewer 9cq6 Q1 ). \n\nSecond, as shown, when evaluated under consistent inference settings (e.g., token limit = 512), our fine-tuned models outperform the original base models across multiple benchmarks. These results confirm that our method achieves genuine improvement rather than merely compensating for LoRA finetuning limitations.\n\n---\n\n**3. On applicability to pre-training and loss coefficient sensitivity**\n\nWe appreciate the reviewer’s insightful suggestions. While our work focuses on post-training due to limited compute and data resources, the proposed losses are theoretically applicable to pre-training. In fact, token distributions in pre-training are more balanced, making it easier to satisfy both load balancing and specialization objectives. We will include this as a future direction in Section 6 (Detail of post-training in Reviewer 9cq6  Q2).\n\nFor hyperparameter sensitivity, we conducted experiments varying α, β, and γ, and adopted a dynamic scaling mechanism to normalize the magnitudes of $\\mathcal{L}_o$ and $\\mathcal{L}_v$ relative to $\\mathcal{L}_{aux}$. This ensures stable optimization across losses. Results show our method remains robust under a wide range of settings.\n\n|$\\alpha,\\beta,\\gamma$|MMLU|GPQA|HumanEval|GSM8K|MATH500|MaxVioGlobal|\n|---|---|---|---|---|---|---|\n|1e-3,1e-3,1e-3|35.59|28.76|43.58|50.94|49.33|2.52|\n|1e-3,1e-4,1e-3|31.24|25.52|41.62|46.63|46.23|3.05|\n|1e-3,1e-3,1e-4|33.52|27.35|39.52|48.30|49.09|2.77|\n|1e-4,1e-3,1e-3|30.74|26.90|42.85|49.62|44.54|4.57|\n\n\nThe default setting achieves the best trade-off between general performance (MMLU) and specialization effectiveness (MaxVioGlobal), demonstrating robustness to coefficient variation.\n\n---\n\n**4. On computational overhead**\n\nWhile the orthogonality loss ($\\mathcal{L}_o$) has a theoretical complexity of $O(N \\cdot k^2 \\cdot d)$, this overhead is practically manageable. In standard MoE setups, the number of active experts $k$ is a small constant (e.g., 6 in DeepSeek models), with $k \\ll N$ and $k \\ll d$. Moreover, the required pairwise projections are highly optimized on modern GPU hardware via batched matrix operations. Therefore, $\\mathcal{L}_o$ introduces limited additional cost and does not become a training bottleneck in practice.\n\n---\n\n**5. On the diversity of evaluated MoE architectures**\n\nWe acknowledge the reviewer’s suggestion to validate our method on more diverse MoE architectures. Our initial experiments focused on DeepSeek and Moonlight models due to their strong open-source performance and recent community adoption, making them credible testbeds for practical effectiveness.\n\nIn response to this concern, we have additionally evaluated our method on two structurally different models: Mixtral and Phi-MoE, which adopt distinct routing strategies and omit shared experts. As shown below, our method continues to outperform baselines across all tasks.\n\n|Method|MMLU|GPQA|HumanEval|GSM8K|Math500|MaxvioGlobal|\n|---|---|---|---|---|---|---|\n|MixtralBasemodel|43.32|15.34|33.23|52.42|20.63|6.32|\n|MixtralAuxOnly|50.56|18.84|39.74|58.73|28.74|3.25|\n|MixtralAuxFree|49.73|20.14|36.06|56.96|29.84|3.58|\n|MixtralOurs|52.63|20.74|37.73|61.74|33.42|3.54|\n|PhiMoEBasemodel|51.73|34.52|66.46|84.52|41.84|7.53|\n|PhiMoEAuxOnly|57.24|34.21|71.32|85.21|42.94|5.21|\n|PhiMoEAuxFree|53.52|35.32|70.45|86.24|44.52|5.35|\n|PhiMoEOurs|59.63|35.87|76.23|88.32|44.79|5.32|\n\nWe will include these results in the revised version to better demonstrate the generality of our approach.\n\n---\n\n**6. On writing issues and limitations discussion**\n\nWe thank the reviewer for the careful reading. \n\n- We will correct the punctuation issue on line 66 in the revised version.\n\n- We will rewrite and expand Section 6 into a more comprehensive *Limitations and Future Work* discussion to improve clarity and completeness.\n\n---\n\nWe sincerely thank the reviewer again for raising many important and constructive concerns. While this review included the most critical questions, we believe we have now addressed them thoroughly in our rebuttal, including the pre-finetuning comparisons, training setup clarification, loss function sensitivity, and generalization to new architectures.\n\nWe would like to reiterate that our paper proposes a novel and effective framework to jointly improve expert specialization and load balancing in MoE models, supported by theoretical guarantees, carefully designed regularization losses, and strong empirical results across 11 benchmarks. The remaining three reviewers have explicitly acknowledged the clarity, novelty, and significance of our contributions.\n\nWe hope our clarifications have addressed your concerns. If you find the revised explanations and results satisfactory, we would greatly appreciate your consideration in recommending the paper for further discussion."}]

Author rebuttal:

Meta-review: (a) Scientific claims and findings

The paper introduces two complementary loss functions—orthogonality and variance—that enhance expert specialization in Mixture-of-Experts models without sacrificing load balance. The approach requires no architectural modifications and yields consistent performance gains across diverse benchmarks.

(b) Strengths

- Addresses a core bottleneck in MoE training (specialization vs. balancing).
- Simple, general, and theoretically well-justified method.
- Extensive empirical evaluation: multiple models, 11 benchmarks, recent strong baselines.
- Demonstrated robustness (hyperparameters, architectures, domain generalization).
- Clear impact for large-scale LLM training practice.

(c) Weaknesses

- Original clarity issues (training setup, dataset sizes, steps) reduced transparency.
- Scope validated mainly for post-training; applicability to pretraining remains open.
- Computational overhead not fully detailed in the first version (later clarified as modest).

(d) Reasons for decision

The contribution is principled, broadly relevant, and highly impactful for advancing scalable MoEs. The rebuttal convincingly addressed reviewer concerns with additional experiments and clarifications, reinforcing the strength of the work. Given the novelty, breadth of validation, and importance to the field, I recommend accept (oral).

(e) Discussion and rebuttal

- Concerns: hyperparameter sensitivity, unclear training setup, limited scope, missing baselines, computational overhead.
- Authors provided detailed clarifications, sensitivity analysis, runtime overhead benchmarks, added recent baselines (ERNIE 4.5, SIMBAL), and results on Mixtral and Phi-MoE.
- Reviewers found the responses satisfactory; all maintained or increased positive scores.
Decision: Accept (oral)