Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
Authors: Ang Lv
Jin Ma
Yiyuan Ma
Siyuan Qiao
Venue: ICLR 2026 Oral (2026)
Keywords: Mixture-of-Experts
Large language models
Auxiliary loss
Expert-router coupling
Expert specialization
Primary area: foundation or frontier models, including LLMs

Abstract: Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities. Our approach treats each expert's router embedding as a proxy token for the tokens assigned to that expert, and feeds perturbed router embeddings through the experts to obtain intermediate activations. The ERC loss enforces two constraints on these activations: (1) Each expert must exhibit higher activation for its own proxy token than for the proxy tokens of any other expert. (2) Each proxy token must elicit stronger activation from its corresponding expert than from any other expert. These constraints jointly ensure that each router embedding faithfully represents its corresponding expert's capability, while each expert specializes in processing the tokens actually routed to it. The ERC loss is computationally efficient, operating only on $n^2$ activations, where $n$ is the number of experts. This represents a fixed cost independent of batch size, unlike prior coupling methods that scale with the number of tokens (often millions per batch). Through pre-training MoE-LLMs ranging from 3B to 15B parameters and extensive analysis on trillions of tokens, we demonstrate the effectiveness of the ERC loss. Moreover, the ERC loss offers flexible control and quantitative tracking of expert specialization levels during training, providing valuable insights into MoEs.

Published as a conference paper at ICLR 2026
COUPLING EXPERTS AND ROUTERS IN MIXTURE-OF- EXPERTS VIA AN AUXILIARY LOss
Ang Lvl:2, Jin Ma². Yiyuan M Siyuan Qiao² RRemmin University of China, GSAI "Bytedance Seed
ABSTRACT
Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities, Our approach treats each expert's router embedding as a proxy to- ken for the tokens assigned to that expert, and feeds perturbed router embeddings through the experts to obtain intermediate activations. The ERC loss enforces two constraints on these activations: (I) Each expert must exhibit higher activa- tion for its own proxy token than for the proxy tokens of any other expert. (2) Each proxy token must elicit stronger activation from its corresponding expert than from any other expert. These constraints jointly ensure that each router em- bedding faithfully represents its corresponding expert's 's capability, while each ex- pert specializes in processing the tokens actually routed to it. The ERC loss is computationally efficient, operating only on n² activations, where n is the num- ber of experts. This represents fixed cost independent of batch size, unlike prior coupling methods that scale with the number of tokens (often millions per batch), Through pre-training MoE-LLMs ranging from 3B to 15B parameters and exten- sive analysis on trillions of tokens, we demonstrate the effectiveness of the ERC loss. Moreover, the ERC loss offers flexible control and quantitative tracking of expert specialization levels during training, providing valuable insights into MoEs.
INTRODUCTION
sottuur--o--xppeettt (MoE, Shazeer al. 2017 Fedus al. 2022 Lepikhin al. 2021 Zoph et al. 2022) is a core architecture in modern large language models (LLMs). In MoE models, the feed- forward layer is split into multiple small, specialized "experts." A linear classifier, known as the "router," selects which experts process each input token. By activating few experts token, MoE balances efficiency with scaled parameter counts, enabling the training of parlameeerr models.
Ideally, a router should possess an accurate representation of each expert's capabilities to enable effective token routing. However, traditional MoEs offer no explicit constraints to guarantee this. Without direct access to expert parameters (and therefore their true capabilities), routers resort to trial-and-error learning of routing strategies, often resulting in misrouted tokens whose gradients interfere with expert specialization. While some methods {Lv et al., 20255 Pham et al. (024) in- corporated all experts' activations for routing guidance, they incur substantial computational and memory costs due to denser activation. A lightweight and effective solution to better couple routing decisions with true expert capabilities remains an open challenge.
We propose expert-router coupling loss (ERC loss), novel auxiliary loss for MoE models that tightly couples routers and experts with negligible overhead. The loss is based on interpreting the router parameter matrix R gaxd cluster centers, where each row R[i] serves as the center for the token set X; routed to expert i. The ERC loss comprises three key steps:
(1) Each Rii is augmented with bounded random noise 5, to obtain R[ii, serving as a proxy for tokens in X, Here, 5; is bounded by half the minimum distance between adjacent cluster centers,
"Work done during the internship of Ang Lv langl v@ www.euucccucccoo at Bytedance Seed.
CCorrsponning author: Yiyuan Ma (mayi yuan, uni www.@yyeddncee.c..oooo com)

---

Published as a conference paper at ICLR 2026
MÜ.nl
Mi.o MILO MIGA
MM.4
R[]]
Rfj
Router parameters cluster centers.
Compute L?
Norm
CERC
LERC
W
W
aM
R(
(I) Perturb R proxy tokens.
R(E]
w0th
M IR
6
RV
R[j] is proxy for tokens routed expert i.
(2) Input all [[]] to all experts.
(3) The ERC loss enforces
Get intermediate activation norms M.
Mii,] < MiG
vi,J
(W) denotes expert i's inpat layer)
M MiL.))
Figure 1: Three steps for computing the expert-router coupling loss.
ensuring that the noise simulates input variations within X, while preventing the crossing of cluster boundaries.
(2) Inspired by prior works (Geva et al. 2021 Liu et al. 2023 Lv et al. 2025) ) the intermediate activation norm serves an indicator of how well capabilities align with the token. We measure the intermediate activation norms of experts that take R(( as input. This step produces matrix
M 1axx, with [i, being the activation norm from expert j given input Rli].
(3) For all + j. the ERC loss imposes penalty wherever the off-diagonal elements M [. or [ i] exceed aM i), where a scalar hyperparameter:
CERC = n²
ΣΣ
(max (Mi - (i.4...0) + max (M (i, O))
Minimizing tightly couples experts and routers through two effects:
Expert specialization: The proxy token R[j elicits the strongest activation from expert i versus all other experts. This indicates that expert is optimized best match the features of its assigned token cluster X,
Precise token routing: Expert is most activated by its designated vector R[i] than to any other R(j] for A This demonstrates that Rid aligns well with the capabilities of expert i, ensuring that the router assigns to this expert the tokens that need most.
We conducted large-scale pre- pre-training experiments models from 3B to 15B parameters, using total of several trillion tokens. The ERC loss not only significantly enhances model performance and narrows the performance gap with competitive yet computationally expensive MoE variant (Lv al. 2023) but also retains the efficiency of vanilla MoEs.
Furthermore, building on the first effect, establish that the ERC loss serves as a powerful tool for studying expert specialization. This property arises from two key features of the ERC loss: (1) the specialization level explicitly controlled by a, and (2) the bound of noise ô, provides a quantitative measure for this level. Through this lens, we reveal trade-off between specialization and model performance. Our findings challenge some beliefs about expert specialization that were derived from small-scale experiments. These quantitative and qualitative analysis methods offer new pathways to advance the understanding of MoE models.
2 BACKGROUND
Mixture-offExperre Our description follows the prevailing SwiGLU structure used by advanced
LLMs (Qwen 2024; DeepSeck-Al 2025 OpenA[ 2025). An MoE layer consists of n experts,
where each expert is parameterized by three matrices: W R W RdxD, and W
2

---

Published as a conference paper at ICLR 2026
RR RDxd_ The layer also includes router with the weight matrix R E R= which takes token a E IR as input and outputs an expert weigh vector:
softmax(2R") R".
Typically, the top-K experts with the highest expert weights are selected to process the token. The processing of æ by expert is given by:
E((x) (SiLU(æW;)) @ (xw;)) w.:
where @ denotes element-wiss multiplication. The final output of the entire MoE layer is the weighted sum of the outputs of the selected experts:
Σ w w Ex()), where & E Top-K(w).
Expert-router coupling via denser activation Autonomy-of-Experss (AoE; Lv et al. 2025) en- codes the routing function into expert parameters. AoE factorizes W, into r-rank matrices W down [Rdxr and W RrxD Each expert processes token up the point after the W down
projection. The expert weight vector is computed using the activation norm at this stage:
softmax (( down for i ......n))
The top-K K experts exhibiting the highest activa- tion norms are selected to continue their forward computation, and the others terminated early. This porm-based selection is justified by the fact that the activation norm of MLPs represents how well their capabilities match their inputs (Geva al. 2021: Liu al. 2023). The computational overhead of AoE scales with the number of tokens during both training and inference. Moreover, this inefficiency worsens as the number of experts π increases or the selection count decreases. Wu et al. (2025) found that using only small, rep- resentative subset of neurons per expert is suffi- cient for "autonomous expert selection," reducing but not eliminating AoE's token-dependent cost.
Used
Piram
Computte 4e B Norm & Softmax
E
Unused Param
447
Router
(a) Mixture-of-Experts
(b) Autonomy-of-Exppe
Figure 2: The overview of MoE and AoE mod- els.
Pham et al. (2024) experts' final output norms supervise router logits. There is infer- ence overhead but the model is fully dense-activated during training, contradicting the core sparsity principle of MoE. Therefore, include it only for background discussion, not baseline.
3 METHOD
After analyzing the strengths and limitations of prior work, we distill three design principles to ensure lightweight, effective, and practically applicable enhancement for expert-router coupling in MoE-LLMs:
(1) Routers must retained in MoE architectures to preserve routing efficiency.
(2) An auxiliary loss that enables interaction between experts and routers strengthen their cou- pling.
(3) The loss must have complexity independent of the number of input tokens and must not introduce activation density beyond that of a vanilla MoE.
Below, we introduce expert-router coupling loss, which fulfills all these principles.
In this paper, "weight" refers to the relative contribution of each expert's output or the strength of the loss function. Please carefully distinguish between "weight" and "parameter."
3

---

Published as a conference paper at ICLR 2026
3.1 EXPERT-ROUTER ROUTER COUPLING LOSS
The expert-router coupling (ERC) loss is motivated by clustering- based interpretation of MoE rout- ing: The routing mechanism in traditional MoE models can be interpreted as a clustering process. where router parameters R E Roxd viewed the router computes an n--imenniioal logit vector representing the weight assigned to each expert. Specifically, the weight for expert i is derived from the inner product between and the cluster cen- ter R(i.. When z belongs to the cluster centered at R(i], this inner product is maximized making expert the top choice.
cluster centers. For any input token RRd
A key advantage of this clustering view is that it enables probing an expert's responsiveness to set of tokens without feeding every token to all experts, unlike prior methods (See $2). Instead, we leverage each cluster center R(i] as a proxy for tokens routed to expert i (denoted as X2), enabling us derive intermediate activations and evaluate how well the expert aligns with proxy token.
Our ERC loss is computed in three key steps:
(1) For each cluster center R(i]. we create perturbed proxy token R
Rlj + Ô o, Rd bounded multiplicative random noise, which we elaborate in $3.2 This noise ensures the proxy generalizes to tokens in X₁ Notably, the perturbed R. is used only for loss computation; routing still uses the clean R to compute router logits, as in standard MoEs.
(2) Each proxy token is processed by the W, parameter of n experts, yielding a total of n² intermediate activations. The L norm of each activation is computed form a matrix M R"*", where M []] corresponds the norm from expert j given input R[li]:
Mi. = R|i] W
(3) To enforce expert-router coupling, for all i and j * i, the ERC loss imposes two constraints, where a scalar E [0, 1] determines their strength:
M[i,]] < aM [
(1)
M < (i,
(2)
Constraint ensures the proxy token R activates its corresponding expert i more than any other expert j. Since tokens similar to R(i are routed to expert i, and given their similarity to R((i), they also elicit stronger activation in expert i than in other experts. This strongest activation indicates that expert is optimized to develop capabilities best suited to X; (Lv et al. 2025).
Constraint requires that expert responds more strongly to its own proxy token R[] than by any other R(j. This ensures each R[i] accurately represents expert i, guaranteeing that tokens most needing expert are correctly routed
As a decreases, the two constraints become stricter, thereby enforcing stronger expert-router cou- pling. Additionally, enables flexible regulation of specialization: smaller a increases the gap between Mi, i] and M ji. reflecting greater expert specialization as experts exhibit more differ- entiated responses to the same inputs. This feature makes the ERC loss a useful tool for investigating expert specialization and provides deeper insight into MoF behavior, as demonstrated in $4.2
We translate these two constraints into expert-router coupling loss, formally defined as:
1 n2
LERC
(max (M - (M. + max (M aM
(3)
The three steps for computing expert-router coupling loss are illustrated in Figure For implemen- tation details, we provide Py TyTrch--ttye pseudocode in Figure &
This assumes all R(i]s have comparable norms. We confirm that the models used in our experiments adhere to this assumption.
4

---

Published as a conference paper at ICLR 2026
3.2 BOUNDED RANDOM NOISE FOR GENERATING PROXY TOKENS
The perturbed proxy token [[] R|: @ ò: makes expert i's coupling generalize effectively from Rii] alone to X. To ensure the perturbed point R[i] remains within its original cluster, require a bounded perturbation. We therefore model the noise 6; as a multivariate uniform distribution, 6, U(1 G, 1 + G,)a j arg ming*4: Φi | Rij"|l be the nearest cluster center. For the noise level e to be sufficient to avoid perturbing the cluster, it must satisfy:
i Rbil
4
(4)
2 R
The derivation of this bound is provided in Appendix We set G, its maximum value, i.e., the right-hand side of this inequality. Notably, the value of is dynamically computed at each layer and every training step.
3.3 EFFICIENCY ANALYSIS
Training efficiency In standard MoE layer, T tokens are processed by K experts, resulting in a total computational cost of DD FLOPs. expert-router coupling loss introduces only 2n Dd additional FLOPs, a cost that is negligible in practical pre-training setups where K is often in the millions. In contrast, AoE introduces an additional overhead of (n K jdr FLOPs (recall that r is AoE's factorization rank; see $2). Given that typical MoE-LLMs operate sparsity levels far below 25% (i.e., n. > 4K), this overhead ratio exceeds r/D, making it prohibitive. A detailed breakdown of the FLOP calculations supporting the above theoretical analysis is provided in Appendix C.1
The efficiency of our method is confirmed in practice. The ERC loss maintains low overhead during LLM pre-training under multiple parallelism strategies, adding only 0.2-0.8% overhead in our ex- periments. We provide complete analysis of these real-world distributed conditions and measured throughputs in Appendix C.2
Overhead-free inference Expert-router coupling loss introduces no additional inference over- head, since is not applied that stage. contrast, AoE maintains the same forward computation. along with its associated overhead.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
We compare the ERC-loss-aupe augleetdd MoE against both the vanilla MoE and AoE baselines. All models trained from seratch with 3B parameters. This parameter size is chosen because it rep- resents the largest scale at which we could successfully train the AoE model under our available resources. Our implementation is based OLMoE (Muennighoff et al. 2025). The models com- prise 12 layers with d = 1536 and D = 768. Each Transformer (Vaswani et al. 2017) layer has 16 attention heads and n 64 experts, where K
experts are selected per token. For the AoE model, we set r = 512 to ensure consistent total parameter count. The number of ac- tivated parameters 500M. Each model is trained on 500B tokens from the open-source dataset do dolmap-v1. 5-sample (Soldaini et al. (024). using a batch size of 3 million tokens. We use the AdamW optimizer (Loshchilov & Hutter 2019) with (2) (0.9, 0.95), a weight decay 0. and learning rate of 4e-4 with cosine schedule decaying to 4e-5. A load balancing loss (Fedus) ct al. 2022) with a weight of 0.01 is applied consistently all experiments.
For simplicity, the loss weight of the ERC loss is fixed 1, and specified.
1 by default if not
We evaluate LLMs on the following tasks: ARC-Challenge CCark al. 2018 Common- senseQA (Talmor et al. 2019). COPA (Roemmele et al. 2011). BoolQ (Clark et al. 2019). Hel-
laSwag (Zellers et al. 2019). OpenbookQA (Mihaylov et al. 2018), SciQ (Welbl al. 2017
Social IQa (Sap et al. 2019), WinoGrande (Sakaguchi et al. 2021), and MMLU (Hendrycks et al. 202
5

---

Published as a conference paper at ICLR 2026
4.2 PERFORMANCE, EFFICIENCY, AND LOAD BALANCING
Figure a) reports the average accuracy across all tasks and task-specific results are presented in Figure It shows that the FRC-loss-augmentte Mož achieves stable performance gains, which significantly outperforms the vanilla MoE and narrows the gap between AoE and MoE.
In terms of efficiency, MoE models with and without ERC loss have nearly identical throughput and memory costs. By contrast, AoE requires 1.6X more training hours and 1.3x higher memory usage, limiting further scaling due impractical training times and out-of-memory issues.
Expert-router coupling loss compatible with the load balancing loss. As shown Figure b)) the difference in load balancing loss between MoE combined with CerC and the vanilla MoE is the order of 10 10-$ This difference is negligible given that the overall load balancing loss magnitude remains around 10 10-2 By comparison, the loss difference between AoE and vanilla MoE is approx- imately 4 x 10 10-4. Although this difference is still small, it is notably larger than the difference exhibited by ours.
(a) Average Downstream Task Accuracy
(b) Load Balance Loss
MoE +CERC
0.0110
56.5
AoE MoE
56.0
0.0108
55.5
0.0106
55.0
0.0104
54.5
54.0
0.0102
300
400
500
100
200
300
400
500
Tokens (B)
Tokens (B)
Figure 3: The 3B-scale MoE with ERC loss achieves substantial, stable gains while maintaining effective load balancing. Figure S shows task-specific details.
4.3 VALIDATING ERC LOSS 5B-PARAMETER MoEs
We scale models 15 billion parameters by increasing n to 256 (keeping K=8) and doubling the model depth. This configuration results in total of 15B parameters with approximately 700M activated. Other training hyper-parameters largely follow the setup Section [.1] As a large-scale, high-sparsity model, the AoE method failed to train due to being overly costly and is thus omitted from comparison.
Table shows that the benefits of the ERC loss persist across various public benchmarks more chal- lenging than those used for 3B models, including MMLU (Hendrycks et al. 2021a), C-Eval (Huang et al. 2023), MMLU-Pro (Wang al. 2024b), AGI-Eval (Zhong al. 2024). BBH (Suzgun al.
2023
MATH (Hendrycks et al. 20216). GSMSK (Cobbe et al. 2021). and TriviaQA (Johhi et al,
2017
The consistent performance improvements demonstrate that method effectively addresses
the expert-router decoupling problem even at scale. Throughout this large-scale training, we ob- served no loss spikes abnormal gradients.
Table 1: Scaling to 15B parameters: ERC loss improves performance on challenging benchmarks.
MMLU
C-Eval
MMLU-Pro
AGI-Eval
BBH
MATH
GSM8K
TriviaQA
MoF
63.2
67.5
31.0 31.9
42.0
44.3
25.7
45.2
47.2
MoE LERC
64.6
69.0
44.2
45.6
26.1
45.8
49.1

---

Published as a conference paper at ICLR 2026
Expert ID
16
24
32
40
48
56
(a) MoE
(b) MoE. LERC
Figure 4: t-SNE projections of W loss provides greater expert specialization.
g in MoE experts trained without and with the ERC loss. Our ERC
4.4 THE ERC LOSS IS AN EITECTIVE TOOL FOR EXPLORING EXPERT SPECIALIZATION
With the ERC loss, experts are more specialized, they exhibit greater discrimination between tokens they process and those they do not, compared to vanilla MoE (without the ERC loss). An intuitive demonstration this specialization comes from visualizing expert parameters. Following Yang et al. 2025). we use t-SNE (van der Maaten & Hinton 2008) to project each row of Wi (where mod = 0) from layer 6 (the middle depth) onto 2D point. As shown in Figure 4 ex- perts in vanilla MoE lack specialization, their parameter features do not form meaningful clusters. By contrast, MoE enhanced with the ERC loss exhibits more distinct clusters, indicating specialized capabilities.
Beyond merely promoting specialization, the ERC loss can also serve as a powerful tool for explor- ing it. We show this capability through two features below.
Feature 1: a enables a controllable investigation into optimal specialization the ERC loss, governs the coupling strength between experts and the router. When 0, the ERC loss encour- ages R[ to be orthogonal to the parameters of other experts, thereby maximizing specialization. Conversely, when a - the loss permits smaller differences in how all experts' responsiveness to R(d,, thus reducing specialization. Notably, = only weakens the ERC loss's 's constraints to their maximum extent; it still retains degree of specialization stronger than the spontaneously emerged specialization in a vanilla MoE model.
Feature 2: E provides a quantitative measure for specialization The noise level e exhibits strong correlation with a, and it can reflect changes in expert specialization throughout the training process. This correlation exists because as a increases, experts are allowed to be more homo- geneous. This growing homogeneity among experts, in turn, reduces the separation between the cluster centers in the router as they are tightly coupled. A smaller separation between cluster centers ultimately derives a smaller e. Thus, is quantitative metric tracking expert specialization.
Experiments The following experiments support these two features. In Figure 5(a),, we plot each training step across a parameter search over E {0.4, 0.6, 0.8, 1.0]. Consistent with our analysis, increasing which reduces expert specialization, indeed leads to a corresponding decrease e. Note that measuring router cluster distance is uninformative in vanilla MoE training without the ERC loss, because the router and experts are uncoupled and cluster distances do not reflect expert capability dynamics. We further compared downstream task performance across different values of a. Figure[5(b) shows that all tested values outperform the vanilla MoE model. This not only confirms the robust effectiveness the ERC loss but also demonstrates that the specialization spontaneously formed by vanilla MoE models is inadequate.
The optimal specialization degree Figure ((b) shows that pursuing extreme specialization is not advisable, model performance degrades with overly striet This highlights trade-off between
7

---

Published as a conference paper at ICLR 2026
(8)
(b)
0.625
56.5
0.600
The value ofe
0.575
Accuracy (%)
56.0
0.550
55.5
0.525
55.0
0.500
0.475
54.5
0.450
54.0
Tokens (B)
100
200
300
400
500
300
400
500
0.4
0.6
0.8
1.0
Vanilla MoE
Figure 5: (a) Since routers deeply coupled with experts, the distance between neighboring cluster centers (i.e., the maximum noise level e) quantitatively reflects changes in expert specialization during training, which is controlled by a. (b) Downstream performance across different values of a.
promoting expert specialization and maintaining effective collaboration, which is under-discussed in previous work.
The optimal specialization degree is influenced by several factors. The core consideration is whether, among all (x) possible expert combinations, an effective K-expert set can be assembled for any given input. in general, smaller values of favor more generalist experts, while larger a higher degree of specialization. However, we currently lack quantitative metrics to characterize "large" or "small" n and K across different models; as result, determining the optimal trade-off remains largely empirical. For example, in our experiments with fixed K = 8: When n = 64, the optimal
support
1, suggesting n 64 is not "large" for 3B-parameter models. In contrast, with n 256, we searched for an optimal a 0.5, indicating n 256 is "large" for our 15B- parameter models. This trade-off is also shaped by other architectural choices, such as the use of shared expert=] A deeper investigation into these interacting factors, reliable quantitative metrics for specialization, and an automated evaluation of the optimal specialization degree for a given model are left as important problems for future works. For practitioners implementing the ERC loss, recommend starting with a = 1, which eliminates expert decoupling and should provide some gains. Further improvement may be achieved by experimenting with lower values, depending the specific configuration of your model.
Several studies (Guo et al. 2025 Liu al. 2024 Hendawy al. 2024) have promoted specializa- tion via expert output orthogonality We argue, however, that orthogonalizing expert outputs does the magnitude (norm) of an expert's response
not equate to achieving extreme specialization, a token remains unconstrained. Moreover, finding a set orthogonalized high-dimensional vectors difficult, making it unclear whether such orthogonality yields sufficiently discriminative rep- resentations. Consequently, one should not interpret these fine-tuning experiments as supporting broad claim that "more specialization is always better." On separate note, orthogonality among router embeddings -RNiEE Team 2025) is only weakly correlated with specialization, since the router and experts are typically decoupled. As demonstrated in ablation studies, enforcing router orthogonality might not be critical factor for pre-training MoE models.
4.5 ABLATION STUDIES
Choice of activations for computing M We considered five candidates for calculating M: ing the norms of (a) RWW g. (b) RW,, (c) RiW, 3). (d) the post-SwiGLU activations (i.e., RRW, RWP).. and (e) experts' final outputs (i.e., (SiLU(RW,)) @ RW, W.). As shown in Figure a). RW, g is the most effective among all alternatives. While using the final output
A shared expert satisfies general input requirements, allowing the remaining experts to be more specialized with the same
8

---

Published as a conference paper at ICLR 2026
(a) Using various activations to compute M
impact
(c) Comparison orthagonalized routers
Marila
Mell
Mali Faed
66.6
66.5
Moll Fope -d
66.5
Vaxila Mud
SLUARt
68.0
56.0
65.5
55.5
56.0
56.0
66.0
64.5
64.6
54.0
54.0
300
400
500
300
400
500
800
400
500
Tokerss (R)
Tokena (R)
Tokens (B)
Figure 6: Results of ablation studies. For detailed task- results, please refer to Figure 9
achieves comparable performance, it incurs a higher cost. We therefore adopt RW, g as our default choice.
Random noise ô enables the generalization of coupling The random noise ö allows R(j to better capture the samples within X, To validate its importance, we conducted an ablation study where we trained an MoE with the ERC loss but removed 6. Specifically, we computed M directly using the original R instead of the noise-augmented R. As shown in Figure removing ô greatly degrades performance. This is because the coupling between routers and experts becomes overfitted R, failing generalize to the real inputs that R[i]s represent.
Comparison with contrastive regularization solely on routers The router orthogonalization loss Saaid-EERNIE-Tee 2025) requires R (the row-wise normalization of R) to satisfy:
R
As shown in Figure ((c), the orthogonalization loss yields only limited gains. We attribute this to our finding that the router embeddings in our baseline MoE model are already nearly orthogonal, with an average absolute cosine similarity of 0.15. This value corresponds to angles between router embeddings mostly ranging from arccos(0.15) imply that all MoEs always have nearly orthogonal router embeddings, this may depend on the data or specific architecture; we report this only as a characteristic of our models, which explains the limited gains from the orthogonalization loss.
81 to arccos(-0.15)) 999. Notably, we do not
This result further demonstrates that weak coupling between routers and experts is more critical issue than imperfect orthogonality in router embeddings. The significant gains from ERC, even when applied to baseline with already near-orthogonal routers, provide clear evidence.
Furthermore, it is important to note that even if both routers and experts are orthogonalized, there no guarantee that each R[i] will be aligned with W Therefore, the ERC loss cannot be reduced to contrastive techniques applied individually to routers or experts, such as orthogonalization loss.
Additional analyses Appendix A provides analyses addressing several frequently asked ques- tions, including the effect of a > 1, and verifying that the model decreases the ERC loss by learning meaningful coupling rather than by manipulating parameter norms.
RELATED WORKS
Auxiliary loss for MoEs Auxiliary losses are crucial for training large-scale MoE models. Most existing work in this area focuses primarily enhancing training stability. For instance, many studies have proposed auxiliary losses to address load balancing challenges (Fedus et al.) 2022. Qiu et al., 2025 Wang al. 2024a); Zoph et al. (2022) introduced the z-loss, which penalizes excessively large logits in the gating network to enable stable training. MoE concepts have also inspired mixtures of attention heads entire layers (Gong et al. 2024
Lin al. 2024). where

---

Published as a conference paper at ICLR 2026
auxiliary losses play critical role in effective optimization. Our ERC loss is the first tailored to strengthen the expert-router coupling. Other related auxiliary losses enhancing expert specialization orthogonality are discussed below.
Expert specialization Dai et al. (2024) introduced a shared expert to handle general capabilities, encouraging the others to more specialized. Guo al. (2025) proposes an auxiliary loss mini- mize the pairwise projections of the selected top-K experts outputs for each token, reducing expert overlap but incurring high cost due K² cosine similarity calculations per token. Other methods scale the number of tiny experts to millions, making each expert more atomic and thus more special- ized (Yang al. 2025; Park al. 2025 He 2024), but are memory- bounded. Beyond efficiency, these methods face three major limitations: (I) no quantitative control over specialization degree; (2) no exploration the specialized- generrllzee ability trade-off; and (3) failure to strengthen expert- router coupling. Our method addresses all three, both efficiently and effectively.
Some works (Guo et al., 2025; Liu et al., 2024 Hendawy et al. 2024) maximize specialization by training orthogonal experts, but their evaluations based on fine-tunile seeettuning (or reinforeement learning) experiments. We contend that orthogonalizing expert outputs is not equivalent to achieving extreme specialization, and further, that the optimal degree of specialization is complex problem affected by various factors and requires further exploration.
Contrastive learning Constraints I and bear similarity to contrastive learning (Chen al. 2020
van den Oord et al.) 2019, Khosla et al., 2020). Some MoE research (Luo et al.] 2024 Guo et al., 2025) applied contrastive learning to expert outputs, encouraging specialization. However, naively applying contrastive learning to either routers experts leaves the weak expert-router coupling unaddressed.
CONCLUSIONS
The weak coupling between router decisions and expert capabilities limits MoE models multiple important aspects. We propose expert-router coupling loss that tightly couples router parameters with their corresponding experts. The proposed ERC loss improves MoE-basee LLMs on down- stream tasks while incurring negligible training overhead. In addition, it exhibits several desirable properties that not only provide deeper insight into the behavior of MoE models but also offer a promising tool for future research expert specialization.
STATEMENTS ON ETHICS, REPRODUCIBILITY, AND LLM USAGE
Our research does not raise ethical issues. For reproducibility, used public data and code, and provide algorithm code in Figure We used LLMs solely for typo checking.
ACKNOWLEDGMENTS
We thank Songhao Wu and Ziteng Wang for their insightful discussions. We are grateful to Ruobing Xie, Yining Qian, and Kaiyi Zhang for proofreading and writing suggestions. Additionally, sin- cerely acknowledge the anonymous ICLR reviewers for their constructive comments and questions, which have greatly improved this work.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Baidu-ERNIE Team. Ernie 4.5 technical report, 2025.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL
2002. 05709
Christopher Clark, Kenton Lee, Ming-Wos Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 2924-22 2936, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 13000 URL bttps://aclanthol..oo org. ootpaattoooooyyooooo/S1-10000
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL htttt::ttta/attts www./ooo....... www.aaaaae.. 05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. 2021. URL https://arxiv.
exq/abs/22 22//2210.1111 14168
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan. Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specializa- tion in mixture-of-experts language models, 2024. /tt.///ttt/////......///////////
06066
DeepSeek-AL. Deepseek-v3 technical report, 2025. URL htap:tpp://aarii..o///////////// 19437
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(10)):1-39 2022. HRRL:ttp://///
org/ httpp//ppppsss/e 211///2100999999 html
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Lan- guage Processing, pp. 5484-5495, Online and Punta Cana, Dominican Republic, November Associatio for Computational Linguistics. doi: 10. 00.18633/1/2221 1166:///000..mmmmmmmmama 446..44.. URL https::/accLanthoooogg org/ 2021. emnlp ma in. 446
Zhuocheng Gong, Ang Lv, Jian Guan, Wei Wu, Huishuai Zhang, Minlie Huang, Dongyan Zhao and Rui Yan. Mixture-of-modulee Reinventing transformers as dynamic assemblies of modules. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20924 209988, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 1033///44////0 main. 1164. URL https: http:///alaaaathcocoooo wwg.///ggreo 20 www.nnnmmaain 1164/
Hongcan Guo, Haolang Lu, Guoshun Nan, Bolun Chu, Jialin Zhuang, Yuan Yang. Wenhao Che, Sicong Leng, Qimei Cui, and Xudong Jiang. Advancing expert specialization for better moe, 2025. https: /// www.i..voooooooooooo..a..oooooo. abs / 25..2555.2222 22323
Xu Owen He. Mixture of million experts, 2024.
URL httpps://axxx.w.orr////////
04153
Ahmed Hendawy, Jan Peters, and Carlo D D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Representations, 2024. htttts://openreviit eww net. www.eeeeooeooromteaaaaeeaaa

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Yuzhen Huang. Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: (0.116533v1/P17-11
URL https: htt.ps::/aclantholooog. org/
P17- P17-11477
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 18661-18673, Curran Associates, Inc.,
2020. URL hatppss/prrooeeenn....rr..s.sss. cepp files/paper/2020 File/ /ttp://666660022222222000022aaa3
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS) hard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference Learning Represen- tations, 2021. URL htt.thtts://// /ttt:::://opereee// ew http://ooummmiddgeeeeerr 7 XHTmYb
Hongzhan Lin, Ang Lv, Yuhan Chen, Chen Zhu, Yang Song, Hengshu Zhu, and Rui Yan. Mixture of in-context experts enhance IIms'long context awareness. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Pro- cessing Systems, volume 37, pp. 79573-79596. Curran Associates, Inc., 2024. doi: 10.52202/ httpppapper_eeepppppaaeeeepeepee Confere-oeeeeeeeeeee pdf
079017-2526.
Boan Liang Ding, Li Shen, Keqin Peng, Yu Cao, Dazhao Cheng, and Dacheng Tao. Diversifying the mfxture-of- representation for language models with orthogonal optimizer, 2024. URL
https://arxivg
09762
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja vu: contextual sparsity for efficient Ilms at inference time. In Proceedings of the 40th International Conference on Machine Learning, ICML'23 23. JMLR.org. 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL htttps:://ppereeeeeeeee net ffr.mmiidmm
Bkg6RiCqY7
Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu Hc, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter guage models, 2024. URLphtts:///axii org. abs 2402. 12851
fine-tuning for large lan-
Ang Lv, Ruobing Xie, Yining Qian, Songhao Wu, Xingwu Sun, Zhanhui Kang. Di Wang. and Rui Yan. Learning, 2025. URLhttpss httttpss://peeeeeeieeeseee hwtp:/wwwwww..eeeeee...... foomm/ -ttpummmmdBBBBBB/CCC.CC
models. In Forty-second International Conference on Machine
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https: :ttttps:///xx/// org/ 100aaaaa//1...00 0809.08.00 02789
12

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig. Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-expene language models, 2025. URL
02060
OpenAl. Gpt-oss series, introducing gpt
2025.
URL https://opeaai.com//ncco
Jungwoo Park, Ahn Young Jin, Kee-Eung Kim, and Jaewoo Kang. Monet:
Mixture of monoseman- tic experts for transformers. In The Thirteenth International Conference on Learning Represen- tations, 2025. URL
net/ Forum?id=10wwl. pp
Quang Pham, Giang Do, Huy Nguyen, TrungTin Nguyen, Chenghao Liu, Mina Sartipi, Binh T. Nguyen, Savitha Ramasamy, Xiaoli Li, Steven Hoi, and Nhat Ho. Competesmoe effective training of sparse mixture of experts via competition, 2024.
Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Demons in the detail: On implementing load balancing loss for training specialized mixture-of-exper models. In Wanxiang Che. Joyce Nabende, Eka- terina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 5005-5018, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 251-0. doi: 10. 12218/6///2222..lll 249.. URL https: //acl///cc/ anthology........ooooooooooooo.m wgw.og...oo....... acl 1an-llon..20 2497
Qwen. Qwen2.5 technical report. arXiv preprint arXiv:2412.151115 2024.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D199 1454. URL https:/ hatp://haaaaaatoooa..ao.oa/ orq/
1494
Noam Shazeer, *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In International Conference on Learning Representations, 2017. URL https:
//openzeview. swpenreview.net/oooroooe wwtp:nneeto.oiiioo ott.://www.....eeo.oooaaae//a...
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether offain-of-thoughhe can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL
13

---

Published as a conference paper at ICLR 2026
2023, pp. 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguis-
tics. doi: 10.18653/w1/2023.... 10..8655w1/00233.mmig. fiidiig.iiiinn-ttt ac1.824. URL httppp///aalaatttoooooooo. Eindings-are -clgss--cl.. 824/
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4149-4158, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 16553//1///199142211
URL htt htt.pss//allanchllooy.oooo
N1 N19-14217
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2019. URL httpt://aaax/ org/ 0abs/1807.0377
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, (866):257--2605, 2008. URL http: jm/r otg/papers/v9/
wanderraatteneBer html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL 2llee3ffee2435444 7deee (ld c4a845aa- Paa.r.pdf
ngs. neurips. ccts.cc/ppee wwtppppr..pepeeeeeeeee fi fssss/pppee//017.
Lean Wang, Huazuo Gao, Chenggang Zhao, Xu Sun, and Damai Dai. Auxillary-loss-free load balancing strategy for mixture-of-experts, 2024a. URL 15664
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01 5744,, 2024b.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsoureing multiple choice science ques- tions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, Septem- ber 2017. Association for Computational Linguistics. doi: 10. 10.18653/1//117-41111 URL httpps//aeeaanthhooorr.oo//////
Songhao Wu, Ang Lv, Ruobing Xie, Xingwu Sun, Di Wang, Rui Yan, and Yankai Lin. Union-of-experts: Experts in mixture-of-experts are secretly routers, 2025. URL https:
Xingyi Yang, Constantin Venhoff, Ashkan Khakzar, Christian Schroeder de Witt, Puneet K. Doka- nia, Adel Bibi, and Philip Tom. Mixture of experts made intrinsically interpretable. In Forty- second International Conference on Machine Learning, 2025. URL htttt:::///ppeeeee/s// net httt/forummiii id- X0P22
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Märquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4890, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 18653/v URL htw.h:tpss.. http:///aaaaattooocco 111.1/1910..../1111111122
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association
for Computational Linguistics: NAACL 2024, pp. 2299-23144 24, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ 1/2024 fin.iiin.n...ss--....... 149. URL https://acl antho ogy. fi
ndi mandings-nan
149/
14

---

Published as a conference paper at ICLR 2026
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL
https://aat //ariv/axvvv... org/ /rrg/bb/// abs/ 2202. 08906
A FREQUENTLY ASKED QUESTIONS
A.1 WHAT HAPPENS IF 1?
Some readers might be interested in the value of at which the ERC loss degenerates to no effective constraints, and the trained model consequently degenerates to vanilla MoE. For bascline MoE, we seek the minimum a that zeros the ERC loss computed from the M matrices of the last checkpoint. Table 2 shows that achieving zero ERC loss across all layers requires pre-trained vanilla MoE baseline. This provides direct evidence that the router-expert coupling in the vanilla MoE is very weak.
in our
Table 2: Post-hoc ERC loss evaluation of the vanilla MoE across a values. For clarity, loss values computed using the original R rather than R. making the results deterministic.
Sonc (a 1)
56.6
Sonc
Value of a
Kanc (at 3)
Layer
Vanilla MoF
2
5
56.0
Accuncy (%)
0
0.87
0.69
0.26
0.00
0.00
55.5
0.42
0.28
0.10
0.00
0.00
2
0.45
0.19
0.00
0.00
0.00
65.0
3
0.25
0.15
0.00
0.00
0.00
0.28
0.08
0.00
0.00
0.00
5
0.24
0.22
0.00
0.00
0.00
54.0
6
0.22
0.15
0.00
0.00
0.00
Tokens
0.21
0.13
0.00
0.00
0.00
8
0.15
0.05
0.00
0.00
0.00
Figure 7: a > 1 leads to degeneration into
vanilla MoEs
9
0.16
0.00
0.00
0.00
0.00
10
0.21
0.09
0.00
0.00
0.00
11
0.50
0.44
0.20
0.20
0.00
We further pre-trained 3B MoE models with the ERC loss at a values of 2 and 3. It is important to note that this experiment is to only demonstrate the effects of loosening the ERC constraints. do not recommend using > in practice, as it contradicts our core motivation: the router and experts will shift from state of no mismatch toward looser coupling constraints, ultimately causing the model to degenerate into vanilla MoE. As shown in Figure the model with 2 yields only limited improvement, while the model with a shows almost no improvement over the vanilla MoE.
A.2 DO MODELS REDUCE ERC LOSS THROUGH MANIPULATING PARAMETER NORMS?
Some readers might assume that simply increasing or decreasing the norms of certain router embed- dings experts will increase the diagonal entries of M, thereby reducing the ERC loss. Below, we (1) explain that any attempt to reduce one term of the ERC loss by manipulating norms will simultaneously increase other terms, and present detailed parameter norms evidence.
15

---

Published as a conference paper at ICLR 2026
Note that M can be written |R[i]] W [cost,j. where cōst, denotes the averaged cosine similarity between R[j] and W I
Increasing [[]l] decreases the following loss term (as the second term increases):
R WR
R[
[ W g cōst,
However, it simultancously increases the following term (as the first term grows):
R W ||oōs;. - |[]]]|| |[ōsj.j.
Similarly, any attempt to manipulate the norms of W to reduce one part of the loss necessarily increases others, assuming that W [t, W)lll remains fixed.
This property ensures that the overall ERC loss is minimized only when the router embedding norms are kept comparable and meaningful coupling established between routers and their experts.
As shown in the first four columns of Table the average parameter norms for models trained with and without the ERC loss are comparable. Mcanwhile, the lower standard deviation under the ERC loss reflects more consistent norms across both router embeddings and experts. In the last two columns the table, we present the ERC loss for each model. The ERC loss significantly higher in the baseline model despite its similar average parameter norms.
Table The first four columns show parameter norms for models trained with and without ERC loss, while the last two show the corresponding layer-wise ERC loss. These results show that MoE + LERC learns a meaningful coupling, rather than trivially minimizing the loss through norm manipulation. All values are evaluated on the last checkpoint.
|[]]]|
LERC Values
Layer
Baseline
+LECC
Baseline
+LECC
Baseline
+LERC
0
1.85++0.39
1.67-+0.31
23.46
24.14+3.00
0.87
0.00
1
0.15
1.13++0.12
30.14:10.68
29.42++0.69
0.42
0.00
2
0.17:4012
1.07++0.09
30.63++0.77
0.76
0.45
0.00
3
1.10:+0.08
1.01+0007
30.18:10.77
29.42++0.78
0.25
0.00
4
1.03++0.08
0.89++0.05
1..59+
29.88 1.09
0.28
0.00
5
0.93++008
0.87+0.06
3..33+11.13
29.86+ 1.06
0.24
0.00
6
0.86++008
0.83+00.07
3..65+1.115
19.82+
0.22
0.00
7
0.82++.07
0.75++0.06
1.25
29.96
0.21
0.00
8
0.77++006
0.76+-0.06
30.46+1.02
29.82++0.8
0.15
0.00
9 0.80
0.74++0.06
3..88
29.86+t0.79
0.16
0.00
10
0.74++0.08
0.69++0.06
30.80+1.03
30.16++0.89
0.21
0.00
11
0.80
0.73++0.10
1..0
31.50 1.26
0.50
0.00
B DETERMINING THE MAXIMUM MULTIPLICATIVE NOISE LEVEL
In what follows, write R; for R( and R for the k-th element of R avoid excessive brackets. ò, is a random vector where each component di, follows a uniform distribution 2/1 e, 1 + e), and all components are mutually independent. The perturbed point is given by:
R (6., (R.,) (R, ,, (R..))
W
[.:6] [
*Formally, cōsi,j ΣRED of W. For clarity, this compact form.
cos O.g.a. where f the angle between Rii and the k-th column
16

---

Published as a conference paper at ICLR 2026
To ensure that R. remains in the same cluster R, it must satisfy:
RR
R RR R2ll?
where j arg +in [
Riill.
Expanding the squared norms on both sides of the inequality yields:
d
R
R
Σ
(6 (R,,
k=1
R. R Σ
R R3,a)²
k=1
Substituting into the inequality and simplifying gives:
Σ & (R,, (R,,a R.,k)) (R R3,,)1 <0.
To ensure this incquality holds for all realizations of 8; we consider the worst-case scenario that maximizes the left-hand side. Define:
A 2R, ( RI,,); B Σ R2 R2,k),
so the inequality becomes:
A B <.
(5)
The worst-case 6, chosen maximize Σ A
8 = 11te if A > 0,
1 - if Ag < 0.
Substituting these values gives:
Σ
As +
A& + <<0.
(6)
k=1
k=1
Now simplify A + B:
Σ Ax + B =2 Σ R - - R Σ R ΣRA,. R2
=2 Σ Ryj -E RR, ΣRG,
R2
- E RS -2 Σ Ru.. 1R
- R, 1
Substituting equation into equation 6 yields:
-I R211 + 2c Σ
(R R.,e)l 0.
Solving for gives:
R - R&
émax 2 25
Ri, (RiKk R
k=
17

---

Published as a conference paper at ICLR 2026
However, computing the denominator of this expression is relatively complex. To balance the effi- ciency of loss calculation, we instead adopt a tighter upper bound for e.
By the Cauchy-Schwand Incquality, the following relationship holds:
d
R R. R
R R.
Thus, we have:
RR R2
R R, R
[R] R. 2 R|
mmax
2 2Σ d k=1 R., R.
R2 R.
The term the right-hand side of the final inequality is the value of we used in the main text. This choice ensures that the perturbed R(i] always remains closer in Euclidean distance R( than to any other Rij + i).
C EFFICIENCY ANALYSIS
Appendix C.I analyzes the ideal FLOPs cost breakdown of the vanilla MoE, as well as the overhead introduced by AoE and ERC loss. Appendix C.2 discusses efficiency with consideration of the multiple parallelism strategies used in real-world MoE pre-training. Both analyses demonstrate the practicality of our method.
C.I FLOPS COST BREAKDOWN OF THREE METHODS
MoE forward FLOP counts:
Each expert in MoE layer involves the following operations, with their respective
Two matrix multiplications of dimension T d with d D, accounting for 4T'D FLOPs. These correspond to the linear transformations parameterized by W, and W,
One dementt-wise multiplication of T x D tensors and one SiLU activation applied to T D tensor. The computational cost of these operations is negligible compared to the matrix multiplications.
One matrix multiplication of dimension T D with d, contributing 2TDd FLOPs. This corresponds to the linear transformation parameterized by W..
Summing these components gives a total of 6T'D FLOPs per expert. For K experts processing T tokens, the total computational cost is therefore 6KTdD FLOPs.
Computational overhead of AoE AoE factorizes the expert matrix Wg E Dxd into low- rank matrices of rank r. To maintain the same number of parameters as the original matrix, we require dr Dr Dd, which gives r d+D
The change in FLOPs compared to MoE is:
T
2ndr
DK
2KDd
All experts Wiwe
K experts W., Top-K K experss original
where T is the number of tokens. Substituting the value of and simplifying leads extra computational cost of:
2T(n - K)dr.
Computational overhead of expert-router coupling loss It introduces n² matrix multiplica- tions, each operating on tensors of shape x and d x D. total, this results in 2n Dd extra FLOPs.
18

---

Published as a conference paper at ICLR 2026
C.2 THROUGHPUTS UNDER MULTIPLE PARALLELISM STRATEGIES
We now assess the overhead of the ERC loss in a realistic large-scale pre-training setup that employs both data parallelism (DP) and expert parallelism (EP). As derived in previous analysis, the computational cost of the ERC loss is equivalent to forward pass on n² //3 tokens. When distributed across devices, the costs are:
Base MoE forward: K-T/ /ddpsiizz
. ERC overhead: (n (n/es ep size)/3
Consider training our 15B-parameter model with the configuration: = 10 = 256, dp_si ze 64, and ep_s ze 8. In this scenario, the ERC overhead constitutes the base model's forward pass cost. This theoretical estimate is consistent with our empirical mea- surements: we observe a throughput of 62.03B tokens/day for the baseline versus 61.52B tokens/day for our model, representing only 0.82% reduction. With a smaller n 64, as in our 3B models trained with dp-size=32 and ep-sizes (i.e., EP disabled), the overhead ratio drops further to 0. 1818 This analysis confirms the practical efficiency of our method.
0.72% of
19

---

Published as a conference paper at ICLR 2026
import torch
import to
import PseudoExpertClass
class McE (nn. (nn.Module):
def _ii__t_it___ (self, args):
()_er)
experexsstss = PseudoExpertClass (args) Shape of experts Ng: (n, d)
seli.R www.ttoo...aaaaaa....... (torch. empty args.n, args.d))
13
self.alpha alpsa
17
def erc_ (self, :):
18
row_diff = (M self.alpha * torch.diag (M) unsqueezee
row_diff_clamped = torch. clam min=0.0)
21
col_diff self.alphe torch. (M) unsqueeze (0))
22
col_diff_clampee torch. torch.clamp clamp min-0.0)
23
mask - torch owwe_nnesseeeeeeee like (M) torch se Revice-M.dev ce)
total_diff (row diff clamped + col ditf clamped) mask
26
27
return total difff.eannnn......
28
def get_noisy_router (self, R);
with porch.no_grad()-
31
norm_R torch.norm(R. dim=1)
32
distances torch.cdist. (R, R, p=2)
distances. fill (fllannaallllltlllaaa
min_dist, torchmmin dim=1)
35
eps min_dist
nerm_
36
37
lew - (1 - eps) unsqueeze())
high (1 eps) (1)
noise rand like_like(R)
40
return (low + noise (high low]]
41
def (sel, x):
erc_less 0.0
45
if sclf.training
46
R solf. get
M torch. norm (torch. (torch.einsum((" ('
self, experts.
, dim=-l)
48
erc loss self.erc_lcast
49
Logits v.view shapeel-111] self.R. logits.
softmax (dim=-1)
torch.topk (scores, dimm-1) expert_weights, expert_indices
53
54
return self exl..experts (x, expert_weights, expert_indices), erc_loss
Figure Pseudo code for expert-router coupling loss in PyTorch.
20

---

Published as a conference paper at ICLR 2026
ARC-Clulleng
Commonsense QA
COPA
Boolj
Tie HallaSSang
Figure
Molt ofanc
OpenBook QAA
SciQ
Social IQA
MMLU (Ang.)
WinoGrrande
Mysf
ARC Challenge
Cocmmousense
COPA
BoolQ
HellaSwag
Figure
OpentBook
SciQ
Social IQA
MML (Avge)
WinoGrande
ARC-Chullane
Commennnne
Buol()
HellaSSagg
Figure 6(b)
Mac Pouac
OpenRock
SeiQ
Social IQA
(Asp)
ARC-Challenge
Commonsennt QA
BecolQ
HellaSSwag
Figure 6(c)
Meli Megac
OpenBook OAA
Sai
Social IOA
MMLU (Aug])
WinooGoadde
AcW
ARC RRC-Challienge
Commonsense QA
COPA
DeelQ
HellaaSwug
Figure 7
OpenPiook QA
Serl
Social IQA
MNI MIILEEA (Avg))
WineClonde
Figure 9: Task-specific downstream results for previous experiments.
21


Reviews:
[{"review_id": "1VzPaO0JWS", "reviewer": "Reviewer_n6Ei", "summary": "This paper proposes a new auxiliary loss for MoE training which better couples the experts and the routers and promotes expert specialization. The idea is to think of the router rows as cluster centers and to generate cluster data from the centers via random perturbations. The data from a given cluster should induce higher activation norm in the corresponding expert relative to the other experts and to data from other clusters. This is enforced by a soft hinge penalty. Adding the auxiliary loss adds modest overhead during training and does not affect inference. The authors find that the penalty improves downstream metrics over a baseline vanilla MoE and is comparable to the more expensive AoE method.", "questions": "In addition to clarification of the weaknesses, I have the following questions:\n\nIs it possible that $\\alpha > 1$ can perform even better? At what $\\alpha$ will you recover vanilla MoE?\n\nWhat about using post SwiGLU activations or $W_o$?\n\nDo the norms $\\lVert R[i] \\rVert$ stay comparable across $i$? If this is not true, then there seems to be a mismatch between Euclidean distance based clustering and inner-product based routing.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The method is simple, intuitive, and clearly presented. The experiments and evaluations are thorough and the auxiliary loss does appear to improve performance.", "weaknesses": "The activation metric is not scale invariant, the auxiliary loss can be decreased in a non-meaningful manner simply by scaling up $W_g^i$.\n\nThe auxiliary appears to make the gradient dense across experts since activations norms for each token are computed across experts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O1XcUq668o", "reviewer": "Reviewer_2vnc", "summary": "In this paper, to mitigate performance degradation caused by the representation mismatch between the router and experts in MoE architectures, each row of the router matrix is treated as a representative vector for the vectors processed by an expert. A constraint is introduced in the form of a loss function, ensuring that when values in the vicinity of this representative are input to the corresponding expert, the activation is maximized. This constraint helps ensure that the expert selected by the router based on the actual input is the one most efficiently activated by that input. Experimental results show that the proposed method consistently outperforms the standard MoE setup, with few exceptions.", "questions": "Regarding the randomness: could a method be devised to make the training behavior more theoretically consistent and predictable? For example, could a derivative algorithm be considered, such as marginalizing $R[i] \\odot \\delta_i$ over $\\delta_i$, or handling the noise as a distribution (without sampling)?\n\nThe proposed method can be seen as a form of contrastive learning between the router and expert features, and therefore it seems relatively natural to consider leveraging techniques from contrastive learning. Are there any thoughts on this at this time?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Many recent MoE papers focus on improving routing. Methods exist to resolve the mismatch between routing and experts, such as adding an auxiliary loss to teach desired properties in expert specialization, or modifying the model architecture, like AoE (which is also adopted as a baseline in this paper). The proposed method belongs to the former category; it achieves its goals regarding expert specialization by simply adding a simple constraint, without modifying the conventional model architecture at all. The proposed method can leverage existing software assets as-is, for example, by being integrated into existing training toolkits. The proposed method is an extremely lightweight loss function, and it is believed to have no or very small practical impact on training speed when introduced into the MoE training process. Personally, I am impressed that the authors conceived of this method, and I would like to try it in our own training framework.\n\nIt is noteworthy that the proposed method enables the router and experts to have an explicit geometric correspondence, succeeding in achieving a similar effect to AoE without requiring significant architectural modifications. It may also facilitate the visual analysis of the model's internals.", "weaknesses": "The experiments only validate the method on a single, very small-scale model instance. It has not been demonstrated whether the method is effective across the wide variety of MoE architectures. Since the experiments involve expensive pre-training, it is understandable that validating on various settings must be forgone due to cost, but it is true that the information provided feels somewhat insufficient.\n\nThe method includes randomness, which may be a source of training instability, although as shown in 4.4 (2), the fact that this randomness contributes to generalization appears to be valid.\n\nA new hyperparameter, $\\alpha$, which is difficult to tune intuitively, is introduced. Given the current lack of experimentation across a wide range of model instances, applying the settings used in the paper directly to other experiments is considered to carry a certain amount of risk.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2N1iNmGFAR", "reviewer": "Reviewer_WsdJ", "summary": "This paper proposes expert-router coupling loss, a lightweight auxiliary loss that couples expert capabilities (activation norm) and the router’s decisions. The authors claim that this loss encourages each expert and each proxy token to match with each other, improving performance.", "questions": "1. Are both R and W_g optimized by ERC loss (rather than R only)?\n2. Can you directly calculate the expectation of ERC loss under \\delta and optimize the expectation directly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The ERC loss is computationally cheap.\n2. The experiments show MoE gets considerable gain from this ERC loss.\n3. Much analysis and ablation are provided.", "weaknesses": "1. You might need to compare with Router Orthogonalization Loss in https://yiyan.baidu.com/blog/publication/ERNIE_Technical_Report.pdf, since your loss is somewhat similar to ||(RW_g)TRWg - I||_F, if you assume W_g^TW_g\\approx I, it is similar to ||R^TR - I||_F.\n2. It seems that this ERC loss can be optimized to 0 when RMS(R) -> 0 or RMS(W_g) -> 0, so will it only serve like weight decay?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)