Multilingual Routing in Mixture-of-Experts
Authors: Lucas Bandarkar
Chenyuan Yang
Mohsen Fayyaz
Junlin Hu
Nanyun Peng
Venue: ICLR 2026 Poster (2026)
Keywords: mixture-of-expert
cross-lingual transfer
multilingual
model modularity
math
interpretability
LLM
model steering
Primary area: interpretability and explainable AI
TL;DR: We provide the first analysis of how MoE LLMs route multilingual texts and this analysis leads to the development of a routing intervention methodology that leads to increased multilingual generalization.

Abstract: Mixture-of-Experts (MoE) architectures have become the key to scaling modern LLMs, yet little is understood about how their sparse routing dynamics respond to multilingual data.  In this work, we analyze expert routing patterns using parallel multilingual datasets and present highly interpretable layer-wise phenomena.  We find that MoE models route tokens in language-specific ways in the early and late decoder layers but exhibit significant cross-lingual routing alignment in middle layers, mirroring parameter-sharing trends observed in dense LLMs. In particular, we reveal a clear, strong correlation between a model's performance in a given language and how similarly its tokens are routed to English in these layers. Extending beyond correlation, we explore inference-time interventions that induce higher cross-lingual routing alignment. We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English, and it successfully increases multilingual performance. These 1-2% gains are remarkably consistent across two evaluation tasks, three models, and 15+ languages, especially given that these simple interventions override routers of extensively trained, state-of-the-art LLMs. In comparison, interventions outside of the middle layers or targeting multilingual-specialized experts only yield performance degradation. Altogether, we present numerous findings that explain how MoEs process non-English text and demonstrate that generalization is limited by the model’s ability to leverage language-universal experts in all languages.

Published as a conference paper at ICLR 2026
MULTILINGUAL ROUTING IN
Lucas Bandarkar' Chenyuan Yang² University of California, Los Angeles
Mohsen Fayyaz
Junlin Hu Nanyun Peng
Fudan University
ABSTRACT
Mixture-of-Experts (MoE) architectures have become the key scaling modern LLMs, yet little is understood about how their sparse routing dynamics respond to multilingual data. In this work, we analyze expert routing patterns using parallel multilingual datasets and present highly interpretable layer- wise phenomena. We find that MoE models route tokens decoder layers but exhibit significant cross-lingual routing alignment in middle layers, mirroring parametcr-sharing trends observed dense LLMs. particular, we reveal clear, strong correlation between model's performance in given lan- guage and how similarly its tokens are routed to English in these layers. Extend- ing beyond correlation, explore inference-time interventions that induce higher cross-lingual routing alignment. We introduce a method that steers the router by promoting middle-layer task experts frequently activated in English, and it suc- cessfully increases multilingual performance. These 1-2% gains are remarkably consistent across evaluation tasks, three models, and 15+ languages, espe- cially given that these simple interventions override routers of extensively trained, of-te-f-the-- LLMs. In comparison, interventions outside of the middle layers or targeting
segeuage-specifff ways in the early and late
experts only yield performance degradation. Altogether, we present numerous findings that explain how MoEs process non- English text and demonstrate that generalization is limited by the model's ability to leverage language-univee universal experts in all languages.
Routing Divergence from English, per Layer [ QWWE3-300--3BB
Mean JS-div (entropy-nomaliieel)
0.30
0.25
Bambara
0.20
0.15
Bengali
0.10
0.05
French
0.00
15
22
29
36
43
48
Layer Number
Our Routing Interventions
Figure 1: Visualization of the typical divergence MoE routing weights across model layers be- tween English and high-, medium-, and low-resource language. There is consistently lower diver- gence in the middle layers, where experts are shared across languages. Languages the model does understand (e.g. Bambara) fail leverage similar experts as top languages. In this work, also present a steering method that activates similar experts to English (red arrows) and results in improved multilingual generalization (e.g. an increase in MGSM-Bengar from 0.776 0.824).
INTRODUCTION
Sparse (MoE) architectures (Shazeer et al. 2017 are the new dominant paradigm
in Large Language Models (LLMs) because they enable tremendous parameter scaling while main-
"Correspondence lucasbandar wwr.ca..r....................a.. ucla..aaa. e.du

---

Published as a conference paper at ICLR 2026
taining manageable inference costs (Artetxe et al. 2022 Du et al. 2022 In terms of interpretabil- ity, MoEs present a trade-off compared to dense models. Their sparsity enables more redundancy in parameterization (Dai et al. 2024 Li et al. 2025) and the routing mechanisms are sensitive and variable (Yang et al., 2025b). However, their discrete expert activation facilitates the analysis of which model components responsible for the end result.
Scaling has driven remarkable progress in LLM multilingual capabilities, yet pre- and post-training are often heavily English-centric. As result, large performance gaps remain beyond select few languages. In recent years, significant effort has been devoted to interpreting the internal mecha- nisms that enable multilinguality. This has generally unveiled shared feature spaces in the middle part of the model, which are pivotal to the cross- lonssllingual transfer of model capabilities (Kojima et al.) 2024 Wendler et al., 2024, Bandarkar et al.) 2025 Wu et al. 2025). However, this work has been limited to dense L.Ms,, whereas sparse activation in MoE architectures leads to different computa- tional structures whose impact on feature representations remains unexplored.
In this work, we investigate multilingual behavior in mixture-of-expert LLMs. To begin with, data analysis comparing routing across parallel datasets yields numerous coherent findings. Studying QWEN3-30B-A31 (Yang et al. (2025a). PHI-3.5 PHI-3.5-MO0 (2025), and OLMOE (Muennighoff et al. 2025), we find that, despite their sparsity. they adopt similar mechanisms as dense LLMs; leveraging language-agnan agnostic parameters in intermediate model layers-if anything, in a clearer, more modular way. In addition, language performance is strongly correlated its cross-lingual routing alignment English. We further highlight how multilingual expert specialization impacts router entropy and token-to-token routing similarity.
MoE (Abdin et 2024), GPT-OSS-20B OpenA]
We build upon this observation by showing that the model's ability call upon shared experts is key driver of multilingual performance. We investigate this via manual interventions into the MoE block's 's forward pass to encourage or discourage the activation of specialized experts. We explore steering the routers in different model layers, intervention strengths, and types of experts. In the end, we find that we can improve multilingual task performance when activating experts important for solving that task in English. We experiment with QWEN3, 10n-3.5-MoE: and GPT-OSS all fully post-trained and tfa L.Mss find that these inference-tine interventions consistently yield statistically significant improvements on two tasks requiring domain knowledge, MGSM (Shi) iet al., 2023) and the medicine subset of GLOBAL-MMLU (Singh et al. 2025). The specific condi- tions under which our intervention works provides strong validation that our initial interpretability analysis uncovered verifiable MoE mechanisms for processing multilingual text.
Through this routing data analysis and the resulting intervention experiments. we demonstrate that improved expert-sharing leads to the generalization of complex capabilities. By demonstrating that simple inference-time interventions yield substantial improvements, our work reveals a vast potential for improving multilingual performance in MoE LLMs. This result motivates the development of other methods that promote cross-lingual expert sharing, such as during training.
RELATED WORK ON MULTILINGUAL LLMS
Before the massive scaling of decoder-only LLMs, smaller encoder-decoder models were subject to the curse of multilinguality, where adding more languages hurt performance in other languages due limited representational capacity (Conneau al., 2020 Pfeiffer al. 2022). Cross-lingual embedding alignment was commonplace with such models is in order to unity feature spaces and facil- itate multilingual generalization (Zhou al. 2016 Schwenk & Douze 2017 Ouyang et al. [02T Patra et al. 2023). But with the shift to decoder-only LLMs, this approach became no longer viable. Nonetheless, these L.MMs implicitly learn shared feature representations. As noted previously, many works have concluded through different approaches that the middle decoder layers of an LLM con- tain joint language representations (albeit English-centric), while the first and last layers primarily map language-specific representations to and from this space (Kojima et al.) 2024 Wendler et al.) 2024 Tang et al.) 2024| Alabi et al. 2024). As model size and training has been scaled, this phe- nomenon has become more evident (Chen et al. 2025) and modular (Bandarkar et al.) 2025). Wu ctt al. (202)) finds that this semantic space extends beyond natural languages, to encompass numbers, computer languages, and different input modalities.
2

---

Published as a conference paper at ICLR 2026
Routing Divergence from English, per Layer
Mean JS-div (entropy-nome woayyoooollliiii malized)
[ Qwen3-30B-A33
12 languages, colored by model understanding
0.40
Belebele 0-shot accuracy
Mandarin
0.30
Bambara
0.25
0.20
as
0.15
Maghreb Arabic
0.05
French
0.00
Layer Number
Figure 2: Visualization with more languages of routing divergence from English across model layers based Qwen3-30B-A3B, where the U-shape can be seen for all. Each line is colored by how well the model understands that language (BELEBELE accuracy), highlighting a strong correlation between the two. We label few notable plotted languages, but provide the same graph (along with 3 more models) colored to better distinguish languages in Appendix A.2
Recent work finds that stronger middle-layer alignment correlates with improved multilingual per- formance (Kargaran et al. 2025 Ravisankar et al. 2025). This relationship appears causal: multilin- gual performance improves when they are explicity steered toward language-shan representations (Mahmoud et al. 2025 Lu et al. 2025) or away from anguage-specingg representations (Lim al.) use of English as a pivot language also boost cross- cross-lingus lingual transfer (Shi et al. [2023] Zhang et al. 2024 Yong et al. (025). By de- sign, [CM team et al. (2024) introduces an LLM that autoregresses over language-neutral "concept" embeddings instead of subword tokens, which exhibits strong multilingual generalization.
2025 Zhao et al. 2025) Prompting methods that encourage
Bridging multilingual research and MoEs, recent works leverage MoE modularity for massively multilingual machine translation (NLBB Team et al. [022 Zhao et al. (024). In LLMs, Zheng al.
(2023) scales multilinguality through MoE upcycling (Komatsuzaki et al. (023) in final layers.
MIXTURE-OF-EXPERE PRELIMINARIES
MoE LLMs differ from traditional decoder-only transformer architectures by replacing the multi- layer perceptron (MLP) component of each model layer with E MLPs, referred to as "experts". For each input token, a router (or "gating network")) calculates a set of logits and sends the token embedding the top-K experts only. The K output hidden states then aggregated, typically via a weighted sum. MoE models are often trained with an auxiliary load-balancing loss (Shazeer et al.) 2017 Fedus al. 2022) that penalizes uneven expert utilization, introducing
redundancy in expert specialization. Token-to-expert mappings are established early in pretraining and are often independent of the surrounding context (Xue et al. 2024).
4 INTERPRETABILITY ANALYSIS
4.1 DATA
For this analysis, we primarily use the FLORES-200 translation dataset (Goyal al. 2022) because of its parallel texts and inclusion of many diverse languages. While no dataset can truly be without domain style, use FLORES and its wide array of topics to represent the baseline, generic do- main. Conveniently, FLORES has an associated reading comprehension evaluation dataset, BELE- tie in language performance. We carefully select
BELE (Bandarkar et al. 2024 which
3

---

Published as a conference paper at ICLR 2026
subset of 12 languages (plus English), diverse in scripts, families, and resource-levels that allow us to explore numerous relationships (See the list in the Appendix A.3)
4.2 MODELS
We look at four prominent open-source MoE LLMs: OLMoE (Muennighoff et al.) 2025 ; OWEN3- 30B-A3B (Yang et al. (2025a), P11-3.5-MME (Abdin et al. (024), GPT-OSS- 20B (OpenAl 2025
All have been trained primarily on English data, but the technical reports of QWEN3, GPT-OSS, and PH1-3.5-M M.. emphasize their multilingual capabilities. Presumably, these three have been pre- and post-trained on significant non- -nnglishh data. Mcanwhile, the older and smaller OLMoE is English- only, exhibiting much poorer multilingual performance. These models all differ in their architectural width, sparsity, and depth. We provide model details and checkpoint specifics in Appendix A.1
4.3 ROUTING DIVERGENCE
We begin by collecting routing data FL.RES for each language across layers. Due to the diffi- culty of cross-lingual token alignment, we average the post-softmax routing weights across tokens to obtain each sequence's 's expert importance distribution. Given language lang and model layer I:
let be the number of experts the Mixture-of-Experts (MoE) layer.
let N be the number of sequences in the corpus.
let Lg be the sequence length (number tokens) of the 7th sequence.
let Pi, (l be the routing weights for the tdh token of the 7t sequence from language lang, at layer (lang.!)
1. This is an E-dimensional probability: If z are the logits, then Pi,
(lang.)) softmax i
(lang.t) 1 M (lang.))
10
qi
Pia
The expert importance q for the 7th sample is the mean-pooled routing weights across tokens (q E 1] and Σq= 1). We consider alternatives, such as averaging discrete activation counts rather than routing probabilities, but these yield sharper, higher-variance distributions that are more difficult to mean-pool. Another option is to only the last token's 's routing weight (or average the last few), as is sometimes done with hidden states. However, routing weights vary more strongly across tokens than hidden states, due factors like part-of-speech, token type, and positional text. Therefore, sequence-wide weight-averaging provides a more stable and representative measure of routing behavior.
For each non- EnngEnglish sequence, we use entropy-normalized Lensen-Shanmen divergence (DH-JS) to compare its expert importance distribution to that of its paired English sequence. Routing entropy consistently decreases across model layers and therefore needs to be accounted for when compar- ing JS diveiveree (a symmetric variant of KL-divergence) across layers. We revisit this trend in Section 4.4 and detail our entropy normalization in Appendix A.4 Finally, we average these diver- gences across all sequences in the corpus to get a metric for routing divergence from English for each layer and language.
Diy Diy (lagg.))
1
Du.Js(
(eng.)) (angs( |g (lamg.0)
E [0,1]
N
This designed metric reveals highly interpretable patterns across languages, models, and layers.
Finding
For all languages, there is much higher routing divergence from English in the first and last layers than in the intermediate layers. The overall trend is this U-shape for all languages (See Figure 2).
And while this trend is the most pronounced and least noisy for QWEN3, this general trend is com- mon to all four models evaluated (See Appendix A.2).. As mentioned, OLMoE has very poor multilingual capabilities and this could explain why the big majority of languages studied do not exhibit this U-shape (See Figure A.2 However, French (fra) and Chinese (zho), high-resource lan- guages it can somewhat process, display this U-shape clearly. GPT-OSS (Figure A.3) displays this trend clearly and is the only one where divergence is higher in the first layers than the last. PHI-3.5- MoE (Figure A.4) displays this trend, but only if you exclude the first two Pirer---Pr-3.5-MM-
4

---

Published as a conference paper at ICLR 2026
perplexingly activates the same few experts in the first layer for all languages. We are unable find an explanation for this, but this could imply very poor load balanninn..g.
This aligns with findings from dense LLMs that reveal language- uniierssl representation spaces in the middle layers LLMs. the representations are aligned, the routing would also be. This means that other factors, perhaps semantic ones rather than lexical ones, determine routing here. Generally, these MoE LLMs have implicitly learned to call upon similar experts across languages. While English is the main pivot language for all models studied, find similar trends when graphing with another focus language. Trends are more flattened if lower-resource languages are used as the focus.
Finding
We find strong correlation between cross-lingual routing alignment intermediate layers and language performance. See Figure[] visualization of this phenomenon on QWEN3.
This figure displays a strikingly strong relationship between a model's ability in a language (line color) and how aligned its routing is with English. This is true across the model layers except the first and last ones. Generally, the highest-resoures languages form the strongest U-shape, with very low routing divergence in the middle layers. For Bambara, example for a language we know the models are all very poor at (near-random BELEBELE performance), the LLMs fail to map its inputs to this semantic space, maintaining high routing divergence throughout. In the middle layers of all models, the correlation coefficient r between the routing divergence from English and BELEBELE accuracy is always strong. For OLMOE, r E [-0.95, -0.80] for all middle layers. Meanwhile, GPT-OSS is the weakest (r E [00.40, -0,00), with P1m13.55 5-MOE and QWEN3 in between. Recall that BELEBELE directly evaluates understanding on those same FLORES passages.
Across all layers, language similarity is also correlated with routing similarity. This is very expected in the first and last layers, where token overlap and structural similarity lends itself to shared param- eterization. Even so, the impact of language families continues into the middle layers where we find that even here, related language pairs like (Bengali, Assamese) or (Romanized Arabic, MSA) have much lower routing divergence between them than unrelated pairs like (Bengali, French) and (Oriya, Serbian). And while this confounds our analysis relating language ability and routing alignment, we find that language relationships only explain a small part of the trends. Visualizing divergence from Chinese, French, or other high-resource languages (instead English) yields very similar plots, al- though the slight attenuation of the trends reflect the English- of the shared representations.
We additionally explore do- mains instead of languages compare. We select AlpaCare- MedInstruct (Zhang 2023) to represent the medical domain,
Mean Routing Entropy, per Layer [ OLMoE-7B-1B ]
al.
GSMSK-Instruct Cobbe et al. 202T) the math- ematical domain, and the English FL.ORES split as the baseline For these domains,
3.75
Moan Routing Entropy
eng
bam
routing divergence from the generic domain exhibits the opposite pattern: higher di- vergence exists in the middle layers (more of n-shape). However, these patterns are weaker, as domains are less different than languages. Nevertheless, this suggests separation of parameterization between multilinguality and task-specific capabilities, as has been observed in dense models (Bandarkar et al. 2025). We revisit this language-task modularity (Choenni et al. 2024 Bandarkar & Peng 2025) in Section 5.2, as it is fundamental to our intervention methodology.
tha
3.25
pes
arb
ary
hin
3.00
2ho
ben
asm
2:50
16
Layer
Figure 3: Routing Entropy per Layer for OLMOE
5

---

Published as a conference paper at ICLR 2026
4.4 ROUTING ENTROPY AND CONSISTENCY
Finding 3
Routing entropy decreases (in other words, routing confidence increases) across model layers all languages, but this decrease happens at much higher rate for non-English languages.
We calculate the entropy of each routing weight distribution and then average across tokens in the datasets. As tokens pass through the model, routers increasingly know which experts to send them to, perhaps because representations become more refined or experts more diverse (Lo et al, 2025). This lowering entropy occurs for English, but is much more pronounced for non-English languages, as can be seen for OLMOE in Figure 3 For non-English languages in particular, the final layer displays major drop. While the entropy graphs look quite different for each model (See Appendix A.5), these broader trends are visible for all. A likely explanation is a small set of experts specialized for non-English generation in the final layers. Meanwhile, English has a broader pool of possible experts to decide from, resulting in higher routing entropy.
We additionally analyze the token-to-token routing variance across languages and layers. To do so, we randomly sample 500 pairs of tokens sequence and take the Jaccard similarity of the two sets of activated experts. This gives robust estimate for the expected similarity across all 2 token pairs. We then average across sequences, giving measure of intra-sequence routing agreement for each layer. We show this metric for 711-3.5-Men with a subset languages in Figure as an example. Other models and languages display very similar patterns.
Finding 4
For non- non-Engloe -unglEnlishhhh languages, there is generally higher routing consistency across tokens within sequence than in English. In the last layers, it is much higher than English.
Routing consistency, itself, also correlates with per- formance, with the lowest resource languages having fra
Routing Consistency Across Tokens, per Layer Phi-3.5-MoE
Mean Intra- Intra-Sequent Sequence Jaceard Similarity
the highest token-to-token agreement. High-resource lan- guages have lower consistency, but still mostly above that of English across model layers. This can be explained by the English-heavy
zho
IIt
pretraining using a load-balancing loss, which would lead to large number of specialized experts for English tokens. In contrast, multilingual tokens rely on fewer experts, resulting in reduced token-to-token vari- ation. In the last layers, models will consistently send tokens the same non- non-Englnn English language to the same expert. These consistency metries further support the existence of many fewer multilingual experts in the later layers than those available for English.
0.0
26
3232
Layer Number
Figure 4: Token routing consistency (within a sequence), across layers in PH1-3.5-MOE.
5 INTERVENTION METHODOLOGY
5.1 EVALUATION TASKS AND DATA
We choose as our target evaluation tasks the multilingual mathematical reasoning benchmark, MGSM (Shi et al.] 2023), and the medicine subset of the multiple-choice benchmark, GLOBAL- MMLU (Singh et al. 2025). Both are fully parallel test sets that require domain-specific knowledge and reasoning, and therefore present good evaluation of cross- lingull ability transfer in comparison to FLORES and BELEBELE, which serve more as pure linguistic signals To identify math domain experts, SSM8K-Innsruce (Chen et al. 2024), which constitutes the training set of GSM8K

---

Published as a conference paper at ICLR 2026
Cobbe et al.) 2021) augmented with instructions. Once again, we use the AlpaCare MedInstruct dataset (Zaagg et al. 2023) to represent the medical domain. We note that GSM8K distribution- ally identical to (English) MGSM, while MedInstruct is only similar to the evaluation data in broad domain. We continue to use FL as the baseline (see Section 4.1
5.2
EXPERT IDENTIFICATION
Nutlingual Experts Mach Experts
Number of Math & Multilingual Experts Identified Qwen3-30B-A39
Number Experts (dentied
Total Experts 48 x 128 6144 Total Nuttilnnusl Total Experts 595 Union = 685
15
Intersection
22 Layer Number
Figure 5: Plot of the Number of Identified Experts per Layer, with 0.3 for QWEN3. The red vertical bars delimit the region in which we intervene. A different view is displayed in Appendix AA8
A.8
Although the goal of Section 4 was to understand layer-wise patterns, the goal here is to identify specialization in individual experts. To do so, weights to better discern the most responsible experts. Similarly to Fayyaz et al. (2026), we calculate the relative frequency of activation, that is the proportion of tokens that expert figured in the top- k, a, L; E F]. While that work uses paired samples, we cannot pair our data across domains and thus average this activation proportion across sequences in the corpus. Then, for each expert, we take the difference A in those averages between the task or language dataset and the baseline. Concretely:
discrete activation counts instead of routing
((1)
A212)
A
a
E[-1,
NO
L
N2
5=1
We find that this method helps clearly identify specialized experts. This is because the resulting As are heavily right-skewed. The large majority of experts are around zero, or right below, while a small percentage of experts have strong positive values (See Appendix A.7 for visualization). These experts, activated much more often on the domain or language dataset compared to the baseline, are therefore specialized. This also shows that, empirically, FLORES is sufficiently "generic" serve as a baseline. To select the experts to intervene on, we use a tunable positive-valued threshold т. Therefore, the hthh expert selected for intervention if A т. Given the high number of languages available, we define multilingual expert as one where for any language A >
Finding 5
There is no overlap between multilingual-speciiliine experts and task (math or medicine) experts.
We find multilingual experts in all layers, but as expected, we find lower prevalence in the middle (See Figure 5 for an example). The degree of specialization is also much lower here (lower Ag- values). Math and medicine experts are more evenly distributed across the model and generally have lower A,-values. If r 2 0.3, we detect absolutely no expert simultaneously specialized for a task and multilinguality across all four models. In other words, the set of experts that are activated more in non-English languages is completely separate from those activated more in the math and medicine domains than the general domain. This is consistent with the routing divergence trends and is very convincing display of language- -aag modularity in MoE LLMs.
This finding provides striking empirical support for the "functional dissociation" of language and thought in LLMs, as proposed by Mahowald et al. (2024). The clear segregation between multi- lingual experts and task-specific experts suggests that the model's internal architecture reflects modular distinction between formal linguistic competence (i.e. in diverse languages) and functional competence (i.e. for a task).
We do investigate with edentifying individual-language experts, but intervention results end up the same.
7

---

Published as a conference paper at ICLR 2026
5.3 ROUTING INTERVENTIONS
The above produces set of experts boost (A or suppress (A)). Then, during the forward pass through the MoE block, our method intercepts the router logits and alters them. This not only has an impact on the sparse activation of experts, but also how heavily each output is weighed during aggregation. Thus, we intervene prior to the softmax operation to not destabilize this weighted sum.
Soft intervention Our first style of intervention simply steers the original logit of the target expert. Because every router produces logits in very different range, we elect to steer by adding/subtracting values (A) proportional to the standard deviation of F. logits, 8(2). Our approach quite different
from Wang et al. (2025), which steers by rather multiplying the weights by factor. Empirically, we
find smaller X S 1.0 work best. Concretely, to steer the thth expert (in A+ or A --):
z - 2k A 8()
(4)
Hard intervention We replicate the force-activation and -deactivation method from Fayyaz et al. (2026), which sets a logit to the maximum or minimum value among all experts given token, forcing its selection or non-selection. We also add a random perturbation e for edge cases (See Appendix A.6 So, if expert & E A+:
4 max(z) +E, E(,100,
(5)
Or rather min(z) i E A-. We note that deactivation is a less significant intervention than acti- vation: deactivation removes one of many options, rather than forcing one of the few activations. Either way, the intervened routing is still far from homogenized across languages. Even hard in- tervention only forces or experts into the top-K selection (where is typically router still produces full logit distribution for every token and selects most experts as before. The intervention is simply a targeted nudge to ensure the task-relevant experts are strongly considered.
8). The
5.4 INTERVENTION EXPLORATION
The search space of possible such interventions is large, given (1) the choice of model layers, (2) the types of specialized experts to target (task or multilingual experts), (3) the threshold that controls expert selection, and finally (4) the direction and (5) strength of intervention. To limit the expo- nentially large search space of model layers (1), leverage our layer-wise interpretability analysis of where strong relationships exist between alignment and (linguistic) performance, i.c., the middle layers. Specifically, we use the divergence graphs (Figure 2 Appendix (.2)) to demarcate which layers constitute this middle zone (See "Target layers" in Table I). For example, we hypothesize (one-indexed).
that boosting alignment would help most in QWEN3'S 's layers 8
We started by exploring the milder deactivation of experts rather than force-activation. As baselines, we deactivated multilingual or task experts in all layers in random subsets, which led sub- stantial performance degradation. Leveraging our hypothesis, we found that significant number of multilingual experts in the middle layers could be suppressed without causing major performance degradation. Similarly, task experts in the early and late layers could be deactivated with minimal negative impact. Deactivating in the exact opposite layers, for each, led to a large drop. While deactivation only led to worse performance, these patterns resonated with hypothesis of implicit language-tase modularity. This informed our subsequent strategy.
Generally speaking, random even slightly suboptimal interventions to the router lead to perfor- mance degradation. This highlights the sensitivity of intervening in heavily-trained MoE routers and the potential over-/under-weite weigh particular experts during inference.
8

---

Published as a conference paper at ICLR 2026
Table 1: Summary of Intervention Results. Target layers are the model layers where the intervention takes place. The expert-selection threshold T and intervention method are described in Section Given the target layers and 7-value, we provide the number of experts selected for steering.
Model name
Total layers
Target layers
(Ak
Interven. method
#Experts selected
Non-Eng AVG
thresh.)
original intervened
MGSM
10 non non-English languages, 250 samples, 2-shot exact-match
QWEN3-30B-AN
48
(8,35)
0.4
x=
22
76.4%
78.0% %
PHI-3.5-MoE
32
(8,17)
0.3
>=
12
57.5%
58.9% %
GPT-OSS-22RB
24
(4,19)
0.3
hard
68.9% 98.9%
71.5% %
Global-MMLU, Medical Subser
non English languages, 420 samples, O-shot accuracy 1
QwEN3-30B-A3B
48
(8,35)
0.5
hard
23
68.2%
69.1 %
PHu-3.5-MoE
32
(8,17)
0.25
soft,
57.8%
58.8 %
GPT-OSS-20B
24
(4.19)
0.3
soft,A=0.5
6
63.8%
64.5% %
INTERVENTION RESULTS
Given the lack of improvement from any deactivation schema, we ultimately turn to the more inva- sive strategy of boosting or force-activating specialized experts, which leads to compelling findings:
Finding 6
For all models and tasks, steering the router to use the same middle-layer experts that it activates for a task in English leads to a statistically significant improvement in multilingual performance.
As discussed, we identify task experts using English in-domain data and the above methodology. Then we intervene to encourage or force the activation of such experts the middle layers when evaluating on multilingual splits of the benchmark. As displayed in Tables [ and this interven- tion method is very consistent in its positive increase for 3 models and evaluation tasks. These gains can be seen across the diverse range of languages, even bit more pronounced for lower- resource languages. While the magnitude modest (1-2 accuracy points), this is substantial given the simplicity of the test-time intervention. It is also statistically significant when considered across languages. The gains for medicine are less pronounced than for math, likely because the dataset for identification is less directly aligned to the evaluation data.
Hyperparameters The models all differed in their MoE configurations (See Appendix and magnitudes of A-values, requiring model-specific tuning for the expert-selection threshold and the strength of intervention (e.g., hard or soft). For soft interventions, |^] = 0.5 tended to be the most successful, as larger X likely disrupted the weights for aggregation significantly. Generally, intervening on a very small number of targeted experts leads to improvements. QWEN3 requires the strictest selection threshold T, while PHI-3.5-MM MOEE works best with minimal interventions (though with 2
each intervention has much greater impact). For GPT-OSS on MGSM, the effectiveness of hard-activation with 0.3 is surprising because here, expert active low as 35% of domain tokens in English would be force-activated on all tokens during evaluation.
Sensitivity to Target Layers Despite the above differences across models, the sensitivity to target layers far exceeds that these tunable hyperparameters for all models. We leveraged our divergence graphs (Figure 2 Appendix A.2) to determine the layers, and using layers slightly outside of this zone led sigmificant performance degradation rather than improvement, This strict delimitation is notable; it suggests that while middle layers are naturally language-agnostic, the routing differences elsewhere represent meaningful specialization that our intervention merely disrupts. addition, these results also validate the efficacy of the visualizations from the routing analysis in revealing these boundaries. Combining with other interventions, such as activating multilingual experts in the top and bottom layers, tends nullify the gains. In the end, experiments show that only by targeting a small number of task-specific experts in these precise, language-univent middle layers can achieve consistent positive gains in multilingual performance.
Overall, the baseline failures and number of conditions under which these interventions do not work highlight the difficulty of intervening in
MoE routers without hurting performance.

---

Published as a conference paper at ICLR 2026
Table 2: Per-Language Intervention Results. Intervention specifies are provided in Table
MGSM
250 samples, 2-shot exact-match
Language
GPT-OSS-20B
stp
QWEN3-30B-A3B
PHI-3.5-MOE
base
intervened
base
intervemed
base
intervened
89.6%
89.2%
(-0.4)
95.2%
94.8%
(-0.4)
88.0%
85.6%
(-2.4)
bn 56.0%
57.6%
(+1.6)
77.6%
79.6%
(42.0)
20.8%
23.2%
(+2.4)
de
69.2%
76.8%
(+7.6)
82.4%
83.2%
(+0.8)
79.2%
81.6%
(+2.4)
76.4%
78.8%
(+2.4)
89.2%
89.2%
(0.0)
85.6%
85.6%
(0.0)
71.6%
71.6%
(0.0)
78.8%
82.4%
(+3.6)
70.0%
69.2%
(-0.8)
ja 77.6%
76.8%
(-0.8)
80.0%
82.0%
(+2.0)
75.6%
77.2%
(+1.6)
sw* 70.4%
76.0%
(+5.6)
78.0%
82.0%
(+4.0)
79.2%
78.8%
(-0.4)
52.4%
62.0%
(+9.6)
48.4%
51.6%
(43.2)
19.6%
20.8%
(+1.2)
te* 71.6%
74.0%
(+2.4)
62.0%
62.4%
(+0.4)
4.0%
4.4%
(+0.4)
58.0%
58.4%
(+0.4)
84.8%
80.4%
(-4.4)
65.2%
68.4%
(+3.2)
zh
86.0%
83.2%
(-2.8)
83.2%
86.8%
(+3.6)
76.0%
79.6%
(+3.6)
non-en AVG
68.9%
71.5%
(+2.6)
76.4%
78.0%
(+1.5)
57.5%
58.9%
(+1.4)
low-res
AVG
60.0%
64.5%
(+4.5)
62.7%
64.5%
(+1.9)
14.8%
16.1%
(+1.3)
GLOBAL-MMLU, Medicine Subset
420 samples, O-shot accuracy
base
intervened
base
intervened
base
intervened
79.0%
78.1%
(-0.9)
82.4%
83.1%
(+0.7)
75.5%
75.5%
(0.0)
58.1% 18.1%
58.8%
(+0.7)
64.5%
64.9%
(+0.4)
55.2%
56.9%
(+1.7)
63.3%
64.7%
(+1.4)
63.6%
63.5%
(+1.9)
39.0%
40.5%
(+1.5)
de
71.0%
70.4%
(-0.6)
75.7%
75.2%
(-0.5)
68.6%
70.2%
(+1.6)
71 11.2%
72.1% %
(+0.9)
76.4%
78.6%
(42.2)
70.5%
71.9%
(+1.4)
fr
71.45 11.4%
71.9%
(+0.5)
77.9%
79.3%
(+1.4)
71.7%
73.1%
(+1.4)
64. 64.0%
63.6%
(-0.4)
66.2%
66.3%
(+0.1)
55.2%
53.1%
(-2.1)
66.7%
67.4%
(+0.7)
75.2%
75.7%
(+0.5)
66.0%
68.3%
(+2.3)
67.1%
67.1%
(+0.0)
76.7%
76.6%
(-0.1)
71.4%
71.4%
(+0.0)
65.5%
67.1%
(+1.6)
72.9%
72.6%
(-0.3)
63.8%
64.5%
(+0.7)
ko 60.7%
61.9%
(+1.2)
68.3%
68.8%
(+0.5)
56.7%
55.7%
(-1.0)
pt 69.3%
69.8%
(+0.5)
75.7%
77.6%
(+1.9)
50.7%
51.7%
(+1.0)
sw
50.2%
51.8%
(+1.6)
43.6%
46.4%
(+2.8)
40.5%
41.9%
(+1.4)
yo 46.2%
47.6%
(+1.4)
42.1%
42.6%
(+0.5)
40.0%
42.9%
(+2.9)
zh
68.3%
68.3%
(0.0)
76.0%
77.1%
(+1.1)
59.8%
60.5%
(+0.7)
non-en AVG
63.8%
64.5%
(+0.7)
68.2%
69.1%
(+0.9)
$7.8%
58.8%
(+1.0)
low-res AVG
53.2%
54.7%
(+1.5)
49.8%
51.5%
(+1.7)
39.8%
41.8%
(+1.9)
As a result, the consistent improvements from our small-scale test-time intervention are notable. The layer-wise conditions under which it happens imply that the inability of such MoE LLMs to activate similar middle-layer task-experts across languages is limitation for cross-lingual transfer.
7 CONCLUSION AND FUTURE WORK
All our analyses of sparse activation patterns converge key finding: the most important multilin- gual specialization of MoE experts occurs early and late model layers, with experts the middle layers serving iaguagg--niveerse mechanisms for multilingual generalization. The strongest idence for multilingual specialization exists in the last layers, where we find lower entropy and high token-to-token consistency in non-English languages. When identifying experts specialized for two other domains, math and medicine, we surprisingly find full separation between the sets of experts for those and for multilinguality. Our manual interventions that steer routers to repli- cate English activation patterns yield consistent multilingual improvements. The specific conditions under which these interventions lead to positive improvement validate our initial analysis and high- light the existence of the model's learned mechanism to attempt to leverage similar experts in the middle layers, even if not always successful. These results suggest causal relationship between cross-lingual routing alignment and cross-lingual transfer. These findings collectively motivate fu- ture work on methods that enhance cross-lingual routing alignment and the sharing of specialized experts, such during training. Additionally, the distinct and modular separability of parameters between language-shir and waaguugge-speccfiee functions suggests opportunities for architectural or training approaches that exploit this natural division.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
All details for reproducibility have been provided in Sections 4 5 and 6 including but not limited to: model checkpoints, evaluation details, formulas, metrics, expert identification method, and inter- vention methods. Certain details have been presented in the Appendix A. A.4 and A.6), but are referenced from the main text. The implemention of the intervention method was done with vLLM (Kwon et al., 2023) and evaluations with the LM Evaluation Harness (Gao et al. (0024).
ACKNOWLEDGMENTS
The authors acknowledge Tanmay Parekh, Sheriff Issaka, and Yunzhi Yao for valuable discussions at UCLA during this work.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck. Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dong- dong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronen Eldan, Victor Fragoso, Jianfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Iter, Sam Ade Jacobs, Mojan Javaheripi. Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weishung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio César Tcodoro Mendes, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez- Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac. Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Yelong Shen, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneetha Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang. Rachel Ward, Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Ji- long Xue, Sonali Yadav, Fan Yang, Jianwei Yang. Yifan Yang, Ziyi Yang. Donghan Yu, Lu Yuan, Chenruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: arxiv. org/
/abgg//////// abs/ 4 04. 14219
Jesujoba Alabi, Marius Mosbach, Matan Eyal, Dietrich Klakow, and Mor Geva. The hidden space of transformer language adapters. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 6588 6607, Bangkok, Thailand, August 2024. Asso- ciation for Computational Linguistics. doi: 10.18853/v a/2024..c URL https:
//aclanthology. wwt.:yy..rg//22224 1e444accll110 acl-long. 356/
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Out, Sam Shleifer, Xi Victo- ri Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang. Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. Yoav Goldberg, Zornissa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 11699-11732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 11/5://///222..oo//// www.:emmmlllaamammm.e.mmmmm 804. URL org 20022
804/
Lucas Bandarkar and Nanyun Peng. The unreasonable effectiveness of model merging for cross- lingual transfer in LLMs. In David Ifeoluwa Adelani, Catherine Amett, Duygu Ataman, Tyler A. Chang, Hila Gonen, Rahul Raja, Fabian Schmidt, David Stap, and Jiayi Wang (eds.), Proceed- ings of the 5th Workshop Multilingual Representation Learning (MRL. 2025), pp. 131-148,

---

Published as a conference paper at ICLR 2026
Suzhuo, China, November 2025. Association for Computational Linguisics, ISBN 979-889977
345-6. doi: 10. 10.18653/w1/2025..o m.1.5mmmmmmmm marl-min..10. URL
mr 1a11mmai..10/
I Lucas Bandarkar, Davis Liang. Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: parallel reading comprehension dataset in 122 language variants. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 749-775, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 118533/1/2024..cllll
URL https: https:://acaaaathhhhaa org/ 2024 211244.cl-oong 447
Lucas Bandarkar, Benjamin Muller, Pritish Yuvraj, Rui Hou, Nayan Singhal, Hongjiang Lv, and Bing Liu. Layer swapping for zero-shot cross-lingual transfer in large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
/ppenevview net for.mmormmmmmm -dh.mmidddvhnw0066
Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. Breaking lan- guage barriers in multilingual mathematical reasoning: Insights and observations. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2024, pp. 7001-7016, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 16263/11222222 http:///ww///22...nnnnnee//////o findings-emnlp.411. URL
https: pettps://aclanthhe ogy org/ 2024. f mndingg emn 1p. 411/
Yuxin Chen, Yiran Zhao, Yang Zhang. An Zhang, Kenji Kawaguchi, Shafiq Joty, Junnan Li, Tat- Seng Chua, Michael Qizhe Shich, and Wenxuan Zhang. The emergence of abstract thought in large language models beyond any language. In The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems, 2025. URL htt.sttts:/// http:://pppeevveiieee net/ forumrirroooo httppss://peeeeeieeeee
ZE 2E9cxnEBpy
Rochelle Choenni, Ekaterina Shutova, and Dan Garrette. Examining modularity in multilingual LMs via language-specializee subnetworks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 287-301, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v122224 findings-naccl.12 na htt.tppps://accaaaoooooo.... 2tt.:/2222......... 2.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1411 2021.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudbary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingua, representation learning scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 1.118653/1/2222.acc-mm 2020.ac1-nan
URL https: / ac ceaeclanhhhoo ogy. org/
747/
Damai Dai, Chengqi Deng. Chenggang Zhao, R.x. Xu, Huazuo Gao, Deli Chen, Jiashi Li. Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y.k. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeekMoE: Towards ultimate expert specialization in mixture-of- experts language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1280-1297, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 00.18653/v1/20244 aclnoo/ URL BBRLttttee//acllaataaooooooooo sel-long.r. 707
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Quoc V. Le, Zhifeng Chen, James Laudon, Anselm Levskaya, Mohammad Norouzi, Adams Wei Yu,
12

---

Published as a conference paper at ICLR 2026
Xiaoqiang Zheng. Jing Li, Yonghui Wu, and Zongwei Zhou. Glam: Efficient scaling of lan- guage models with mfxture-of-cxperss. In International Conference on Machine Learning, vol- ume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMLR, 2022. URL
mll...mmrr. press/ 322/du2cc.hn html
Mohsen Fayyaz, Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Ryan Rossi, Trung Bui, Hinrich Schütze, and Nanyun Peng. Steering moe llms via expert (de)activation. In The Fourteenth International Conference on Learning Representations, 2026. URL : 9V8rJs
//openreview.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1), January 2022. ISSN 1532-4435.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
/tt////eoooooo//oo//// org recor ds 12608602
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, San- jana Krishnan, Marc Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 eval- uation benchmark for low-resource and multilingual machine translation. Transactions of the doi: 1A.1166//acla.004477 URL org/ 2022. (ttp:/2022.taac1111 tact-1.3111...1
Association for Computational Linguistics, 10:522-538, 2022.
Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, and Hinrich Schuetze. MEXA: Multilingual evaluation of English-centric I.LMs via cross-lingual alignment. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Findings of the Association for Computational Linguistics: ACL 2025, pp. 27001-27023, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256- http:///./2.2....nnn-nneee findings-acl... 1385. URL https: /accaaaaaaaa pttttts:s//cllatthclleeeo Findings-aci 1385/
5. doi: 10. 10.18653/w1/205,
org/2025.
Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, and Yutaka Matsuo. On the mul- tilingual ability of decoder-based pre-trained language models: Finding and controlling language- specific neurons. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin- Human Language Technologies (Volume 1: Long Papers), pp. 6919-6971, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/w1/2024, naacl long.
guistics:
naacl-long.384,
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. In The Eleventh International Conference Learn- ing Representations, 2023. WRLhtttss: http::::///ppeeevieeeee net/ /tt://ormmmiddnnnnnn
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
LCM team, Loie Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R. Costa-jussă, David Dale, Hady Elsahar, Kevin Heffernan, Jo ao Maria Janeiro, Tuan Tran, Christophe Ropers, Eduardo Sánchez, Robin San Roman, Alexandre Mourachko, Safiyaah Saleem, and Holger Schwenk. Large Concept Models: Language modeling in sentence representation space. 2024.
URL pttttp:://axiiii
/222//24122 08821
Junzhuo Li, Bo Wang, Xiuze Zhou, Peijie Jiang, Jia Liu, and Xuming Hu. Decoding knowledge attribution in mixture-of-experts:: A framework of basic-refinement collaboration and efficiency analysis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar
13

---

Published as a conference paper at ICLR 2026
(eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguisties
(Volume 1: Long Papers). pp. 22431-22449 Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979-8-89176-25111 doi: 1/.18653/1/2222..alll 1093. URL 1093/
https: atthh/hlllattttoccoooo http:///c....oooooooooooor////..
Zheng Wei Lim, Alham Fikri Aji, and Trevor Cohn. Language-eeeeeeeeccciice latent process hinders cross- lingual performance, 2025. URL http:/tt//////x/.oo///////// abt/ 13141
Ka Man Lo, Zeyu Huang. Zihan Qiu, Zili Wang. and Jie Fu. A closer look into mixture-of- experts in large language models. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 4427-4447, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8-89176-198 7. doi: 10. 10.18653/v120255 (ing:222.iiinninn-naa.....
TRLhttps:: haaaacaaanthlllaaa ogy. oggg:lg/00..0052....
Findings
Bacnnngnnnaallcc www.aaanc....mmm .211/
Meng Lu, Ruochen Zhang, Carsten Eickhoff, and Ellie Pavlick. Paths not taken: Understand- ing and mending the multilingual factual recall pipeline. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Con- ference on Empirical Methods in Natural Language Processing, pp. 15066-15096 Suzhou, China, November 2025, Association for Computational Linguistics. ISBN 979-8-89116-332. 6. doi: 10. 10.18653/v1/2022 wtt.:025..mmmmmmmmm6m0m660 URL https: httpe/////aaaatthotogoooo org.22225 emn 1 p-ma in. w 762/
Omar Mahmoud, Buddhika Laknath Semage, Thommen George Karimpanal, and Santu Rana. Im- proving multilingual language models by aligning representations through steering, 2025. URL
fitps://arxiv: orgg
abb/ abs 2505. 12584
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 28(6)5117-540, 2024. ISSN 1364-6613. doi: https://dol.og//0..111jj.i
2024.01.011. URL 81364661324000222
/www http://wwwwccinnneeeeee..tt htt.://ccecctcccooccccccc//e///e http://rr....ccccceeeeeeeeaaaaaa
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Evan Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Deumers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanprect Singh, and Hannanch Hajishirzi. OLMoe: Open mixture-of-eeppets language models. In The Thirteenth International Conference on Learn- ing Representations, 2025. httph:
net/ forum pewrrummii=xxxx dww.:
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loie Barrault, Gabriel Mejia-Gonzalezz Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machine translation. 2022.
OpenAl. gpt-oss- 120b & gpt-oss-20b model card, 2025. URL https:://rxxv..ogg/// 2508. 10925
Xuan Ouyang. Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang- ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with mono- lingual corpora. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 27-38, Online and Punta Cana, Dominican Republic, November 2021. Asso- ciation for Computational Linguistics. doi: 18.18653/v 1tt/33/112222.emm h2t0///.emmmlmmmmii..... URL https:
7/acl antho wgw.ool..o.oooo.........o....... org/2021 emn 1ppmaiinn33
14

---

Published as a conference paper at ICLR 2026
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaud- hary, and Xia Song. Beyond English-centric bitexts for better multilingual language represen- tation learning. In Anna Rogers, Jordan Boyd-Graber. and Naoaki Okazaki (eds.), Proceedings of the s Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15354-15373, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10. 116533///2023.acllooong acl Long. 8567
URL https:
org/22233
Jonas Pfciffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe.
Lifting the curse of multilinguality by pre-training modular transformers. In Marine Carpuat, Cathe-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Techmologies, pp. 3479-3495, Seattle, United States, July 2022. Associa- tion for Computational Linguistics. doi: 10. 10018666/1112222.aaall mare22.aacl-mmn mallmmmiin...... 7/ 1/aclanthologe orq/ 2022. wwwcc-aaaaaaaaaa mww.maainn.mmm 255/
URL https:
Kartik Ravisankar, Hyojung Han, and Marine Carpuat. Can you map it to english? the role of cross- org/
lingual alignment in multilingual performance of Ilms, 2025. URL
abs/2504 09378
Holger Schwenk and Matthijs Douze, Learning joint multilingual sentence representations with neural machine translation. In Phil Blunsom, Antoine Bordes, Kyunghyun Cho, Shay Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih (eds.), Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 157-167, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 18653/1/ W17-2619. URL htt.htts::/// http:///aacaaaattooooyoooooe /ttp///1122661 7-2619/
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. URL pettps://openreeee
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Lan- guage models are multilingual chain of-thoughtt reasoners. In The Eleventh International Confer- ence on Learning Representations, 2023. URL pettps://openreeee httprr/rwwwee.eeefoouuir........ forttfformmidd IXpp
net
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwatt Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng. Shayne Longpre. Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzich Fadaee, Beyza Ermis, and Sara Hooker. Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mo- hammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 18761- Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN 2025 11255acl-long.919
doi: 10.18653/v1/ long. 919/
Tianyi Tang. Wenyang Luo, Haoyang Huang, Dongdong Zhang. Xiaolei Wang. Xin Zhao, Furu Wei, and Ji-Rong Wen. Language-speciffc neurons: The key to multilingual capabilities in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5701-5715, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10. N..11865/v112000 acl-oog.33 URL htthhhtts//eaet an orggy.rrg//02.. acl www.ong.300 3097
Mengru Wang, Xingyu Chen, Yue Wang. Zhiwei He, Jiahao Xu, Tian Liang, Qiuzhi Liu, Yunzhi Yao. Wenxuan Wang, Ruotian Ma, Haitao Mi, Ningyu Zhang, Zhaopeng Tu, Xiaolong Li, and Dong Yu. Two experts are all you need for steering thinking: Reinforcing cognitive effort in moe reasoning models without additional training. In The Thirty-ninth Annual Conference on
15

---

Published as a conference paper at ICLR 2026
Neural Information Processing Systems, 2025. UBL.Bttps//oeeneeeiew...o
Id=x7fCiuCCAu
Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Do Ilamas work in En- glish? on the latent language of multilingual transformers. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pp. 15366 15394, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.
URL
www.:rg//2222...2 8207
Zhaofeng Wu, Xinyan Velocity Yu, Dani Yogatama, Jiasen Lu, and Yoon Kim. The semantic hub hypothesis: Language models share semantic representations across languages and modalities. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
Cooeerevvew..net/oriinne
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You. Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv prepriní 2rXiv:2505.09388, 2025a.
Xingyi Yang. Constantin Venhoff, Ashkan Khakzar, Christian Schroeder de Witt, Puneet K. Dokania, Adel Bibi, and Philip Torr. Mixture of experts made intrinsically interpretable. In Forty-second International Conference Machine Learning, 2025b. URL https:// wwwpurrrieew.ee//orummmmaaa
Zheng-Xin Yong, M. Farid Adilazuarda, Jonibek Mansurov, Ruochen Zhang, Niklas Muennighoff, Carsten Fickhoff, Genta Indra Winata, Julia Kreutzer, Stephen H. Bach, and Alham Fikri Aji. Crosslingual reasoning through test-time scaling, 2025. URL. https:: http:::://rxxvv.orggge www.xv.oorggab////
$5555004000
Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacare: Instruction-tuned large language models for medical application, 2023. URL https:
/ //rxivvorg//bs//2303 2211//55111111119
Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. PLUG: Leveraging pivot language in cross-lingual instruction tuning. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting the Asso- ciation for Computational Linguisticy (Volume 1: Long Papers), pp. 7025-7046, Bangkok, Thai- land, August 2024. Association for Computational Linguistics. doi: 10.11655//1/202..aclh 379. htthhttss:/ /tt:///cllaa http::/:s//.laaaooooooo.oo.o www.oo/////2..1111.............. 3799
Weixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu, Wenxuan Zhang, Yulin Hu, Xingyu Sui, Yanyan Zhao, Wanxiang Che, Bing Qin, Tat-Seng Chua, and Ting Liu. When less language is more: Language-reasoning disentanglement makes LLMs better multilingual reasoners. In The Thint--inth Annual Conference on Neural Information Processing Systems, 2025. URL net 1et
Xinyu Zhao, Xuxi Chen, Yu Cheng, and Tianlong Chen. Sparse moe with language guided rout- ing for multilingual machine translation. In The Tweifth International Conference on Learning Representations, 2024. URL https: http///pprrvviiieee
16

---

Published as a conference paper at ICLR 2026
Guorui Zheng, Xidong Wang, Juhao Liang, Nuo Chen, Yuping Zheng, and Benyou Wang. Ef- ficiently democratizing medical LLMs for 50 languages via a mixture of language family ex- perts. In The Thirteenth International Conference Learning Representations, 2025. URL
https: / http////ppeeeeeee.eeeeeee....ee. net / forum? dd 7idSUU
Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. Cross-lingual sentiment classification with bilingual document representation learning. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1403-1412, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18633/v 1 1133. URLhttpp acc antho www.annwww.orrgoooooooo ogw. 6- 1133/
17

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 MODEL. DETAILS
Table A.1: Details of the MoE LLMs Discussed
Model
Total Params
Active Params
Model Layers
Num. Experts
Active Experts
Checkpoint Cita- tion
OLMOE
7B
1.0B
16
64
8
OLMoEE-1B-7B -mmmm-l--m
Muennighoff 2025
PHI-3.5-
42B
3.8B
32
16
2
Phi-3.5-Mon Mos-unstruct
MOE
Abdin et al.
2024
GPT-OSS
20B
3.6B
24
32
4
gpt- 20B
OpenAl
2025
QWEN3
31B
3.3B
48
128
8
Qwen 30B-A3B
Yang et al.
(222aa)
A.2 ROUTING DIVERGENCE PLOTS ALL MODELS
See Appendix A.3
for language code mappings.
0.50
Routing Divergence from English, per Layer Swwen3330--3B--
fra
0.45
Mean JSD Entropy-normalized
0.40
ary
0.35
zho
spp
It
0.25
ary
0.20
asm
0.15
0.30
0.05
0.00
Layer Number
Figure A.I: Mean
JS-Div per OLMoE layer for 12 non-Engn non-English languages. This is the same plot Figure simply colored for language labeling.
18

---

Published as a conference paper at ICLR 2026
0.55
Routing Divergence from English, per Layer [LMoE-7B-11B]
bem
tha
Mean JSD (entropy-normllite
pes
0.40
hin
0.35
0.30
l
0.25
bert
0.20
asm
0.10
0.05
0.00
16
Layer Number
Figure A.2: Mean entropy-normalizz JS-Div per OLMOE layer for 12 non-English languages. We note OLMoE's poor multilingual capabilities. For languages OLMoE does handle reasonably, French and Chinese, the U-shape is still visible.
0.55
Routing Divergence from English, per Layer GPT-OSS-20B
bam
Mean JSD (entropy-nommil nommaizee)
0.50
fra
0.45
pes
0.40
ary
hin
0.35
zha
0.30
B
0.25
ary
asm
0.15
0.10
0.05
0.00
Layer Number
Figure A.3: Mean entropy-normalized JS-Div per GPT-OSSlayer for 12 non-English languages.
Routing Divergence from English, per Layer Phi-3.5-MoE)
bern
0.55
Mean JSD intropy-normalizeer
0.50
tha
0.45
0.40
hin
0.25
hen
0.20
asm
0.15
0.10
21
76
31
Layer Number
JS-Div per PHI-3.5-MoP layer for 12 non-English lan- guages. Compared to the others, PH1-3.5-MOE does not display the same U-shape, as the first few layers surprisingly have very low divergence, especially the first layer. We verified and saw that a small subset of available experts were being called for all languages in layer which behavior that requires further investigation.
Figure A.4: Mean
19

---

Published as a conference paper at ICLR 2026
A.3 LANGUAGE CODE INDEX
Table A.2: The FLORES codes are used by FLORES and BELEBELE, while the 2-letter codes are used by MGSM and GLOBAL-MMLU, Our plots display the 3-letter code for brevity. We omit some abbreviations in order clarify which languages were used in which component of this work.
FLORES Code
Language
Script
2-1etter
3-letter (analysis)
(evaluations)
arb urrb Beng
Modern Standard Arabic
ar Arab
arb
Assamese
Beng
asm
Latn
Bambara
Latn
bam
ben ben_Beng
Bengali
Beng bn
ben
deu Latn
German
Lain de
eng Latn
English
Latn en
spa_Latt
Spanish
Latn
Lra_Latn
French
Latn fr
fra
Deva
Hindi
Deva hi
hin
Latn
Indonesian
Latn
Latn
Italian
Latn it
jpn _pan
Japanese
Jpan ja
kor Hang
Korean
Hang
ko
Lat_Latn
Lithuanian
Latn
lit
ory-Orya
Odia
Orya
ory
pes_Arab
Western Persian
Arab
pes
por Latn
Portuguese
Latn
pt
rus Cyrl
Russian
Cyrl
srp. eyrCyyl
Serbian
Cyrl
srp
swh Latn
Swahili
Latn sw
tha_Tha
Thai
Thai
tha
telTelu
Telugu
Telu te
yor Latn
Yoruba
Lain yo
zho Hans
Chinese (Simplified)
Hans zh
zho
A.4 ENTROPY- NOOOPAAATOOMAI
Our decision to normalize our divergence metrics comes from the strong per-layer entropy patterns we see in the models (See Section 4.4 and Appendix A.5). KL-divergence, sometimes referred to as "relative entropy", highly sensitive on entropy and therefore so IS-divergence. With entropy decreasing across model layers near- moattooott...ally.. using simple JS- JS-divergence meant our plots were overshadowed by the trends entropy and therefore choose to control for it. This divergence metric properly compares divergence between scenarios when both are flattened distributions (high entropy) or peaked distributions (low entropy).
We normalize JS-divergence by approximating the theoretical maximum divergence given the en- tropy of the two distributions. The normalization factor F is computed as log E Havg- where E is the vector size (number of experts) and Harg is the average entropy of the two distributions. This normalization accounts for the fact that distributions with different entropy have different upper bounds for JSD. Concretely, using the same notation Section 4.3)
1
(as(g (gg
(Dis
+ DKT. (t) (1) where q (q (1) + (2))
(6)
2
F log E (H(q(2)) (()
Du (1) (1 (2) -
Das(q (2)
As reminder, E. is the size of (1), (2) and corresponds the number of experts.
20

---

Published as a conference paper at ICLR 2026
A.5 ENTROPY PLOTS FOR ALL MODELS
Mean Routing Entropy, per Layer OLMoE-7B-1B
Routing Entropy, per Layer Qwen3-30B-A3B
61g
2.50
Vear
210
2.5
07
Figure A.5: Entropy per layer, QWEN3
Leyer
Figure A.6: Entropy per layer, OLMoE
Mean Routing Entropy, per Layer (hi-3.5-MoE]
Routing Entropy, per Layer GPT-OSS-20B
3440
Vean
Ween
5403
Figure A.7: Entropy per layer, PHI-3.5-MoE
Figure A.8: Entropy per layer, GPT-OSS
A.6 MINUTE DETAILS FOR HARD INTERVENTION
Similar to Fayyaz et al. 2026), we guard against the potential of breaking the top-k logic by adding random perturbation in Equation 5 In the where the the number of experts selected for force- activation is not less than the model's experts-per-tokeer (k), the LLM would otherwise throw an error trying to pick experts from > k + 1 experts with exactly the same maximum value (at least with our vLLM implementation). Tiny random perturbation ensures the values are not identical and the top-k can be chosen. We note, however, the activation the experts are not technically guaranteed under this hard-intervention. However, this ends up being inconsequential that hard-intervening on so many experts at once derails the model anyways. This even holds true for 201-3.5-MoBB where only 2 experts are activated.
empirically, find
21

---

Published as a conference paper at ICLR 2026
EXAMPLE PLOT FOR DIFFERENCE RELATIVE FREQUENCY OF ACTIVATION A.7
Diff in GPT-OSS Expert Activation (Swh-Eng) Parallel Math Data, Layer 15
Difference relative frequency (SW) minus EN)
0.2
0.1
0.0
Experts
Figure A.9: Example of visualization our metric selecting specialized experts. This the A described in Section 5.2, the difference in activation relative frequency defined in Equation 3 Here we display an example; A between Swahili and English (on FLORES).
Positive values mean the expert is activated more in Swahili than English. Since the distribution has to be mean-zero, we see most experts have slightly negative value while small few (in this case 3) are strongly positive. This type of graph was almost always the case, across models, model layers, languages and domains when comparing to FLORES English set. This allowed for clear selection of language- or domain spmain--pecialized experts using the threshold
A.8 LOCATION OF SPECIALIZED EXPERTS
Number of Math & Multilingual Experts Identified (Qwen3-30B-Al
Math Multilingual
44
Layer Number
24
09
Expert Index
Figure A.10: A different view from Figure of the location of identified experts, displaying the lack of overlap. Here for QWEN3, and expert numbers are an ID, it is numerically meaningless; simply meant to display the sparsity. The red horizontal bars delimit the region in which we intervene.
0.3 used for identification. Layer numbers are from bottom top
22

---

Published as a conference paper at ICLR 2026
Number of Math Multilingual Experts Identified [GPT-OSS-20B]
Multilingual
Layer Number
26.
12 166
20
24
28
32
Expert Index
Figure A.11: Now for GPT-OSS-201 also with T 0.3 used for identification. Compared to QWEN3, GPT-OSS had less layers and less experts per layer but the sparsity is similar. Regardless, there is an intersection of between the math-specialized experts and 63 multilingual-specialln
experts.
23


Reviews:
[{"review_id": "J1bVHF6NLb", "reviewer": "Reviewer_rfTm", "summary": "The paper investigates multilingual behavior in MoE LLMs. Using models such as Qwen3-30B, Phi-3.5-MOE, GPT-OSS-20B, and OlmoE, the authors analyze expert routing patterns across languages and layers. They find that early and late layers are language-specific while middle layers exhibit cross-lingual alignment. They propose inference-time “routing interventions” that encourage the activation of experts often used for English, reporting 1–2% improvements on multilingual benchmarks (MGSM and Global-MMLU).", "questions": "- The findings that early and late layers are language-specific, while middle layers exhibit cross-lingual alignment, also hold for dense-model architectures, as noted by the authors. You may want to reference this recent paper on the topic: https://aclanthology.org/2025.findings-acl.1385\n- Overall, my assessment is toward acceptance, as it is a timely topic.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Timely topic: Multilingualism in MoE architectures is underexplored; the paper touches on a relevant question for LLM scaling.\n- Comprehensive dataset use: Employs multiple evaluation datasets (FLORES, BELEBELE, MGSM, Global-MMLU).\n- Interpretability focus: The routing-divergence visualizations and entropy analyses are informative.\n- Reproducibility: The methodology is relatively transparent, with sufficient detail to replicate the experiments.", "weaknesses": "- Marginal contribution: The reported 1–2% improvements are minor and lack depth of analysis. No theoretical insight or convincing causal mechanism is demonstrated. The “steering” interventions are heuristic and depend heavily on ad hoc hyperparameter tuning (λ, τ, and layer ranges). No systematic ablation or generalization evidence is provided.\n\n- Interpretation overreach: The claim of “causal relationships” between routing alignment and multilingual performance is not empirically supporte (correlation is misinterpreted as causation).\n\n- Lack of broader impact or insight: The paper neither improves multilingual modeling methods nor contributes significant interpretability tools.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKhN9zmgyc", "reviewer": "Reviewer_gR2T", "summary": "Paper presents a descriptive analysis of how MoEs process multilingual data and investigates the router patterns of tokens for different languages and presents the trends like tokens are routed in language-specific ways in the early and late layers, but exhibit significant cross-lingual alignment in the middle layers, mirroring phenomena observed in dense models. The paper's core contribution is demonstrating a strong correlation between a model's performance in a given language and the similarity of its token routing to English in these middle layers. Paper then presents a simple intervention to leverage this phenomena to sterr model to activate English-identified task-specific experts for non-English inputs, however this method yields only a marginal boost in performance.", "questions": "1. The paper suggests a causal link between routing alignment and multilingual performance, but the evidence feels somewhat weak. A stronger case would involve showing that performance systematically improves as the degree of steering increases, or that steering towards unrelated experts consistently hurts performance. Is there enough evidence to make a strong causal claim here?\n\n2. In Table 2, the intervention appears to slightly degrade performance in English. How do you interpret this? Could it be that forcing a fixed subset of experts interferes with the model’s naturally optimized routing for its primary language?\n\n3. Regarding Finding 5, the clean separation between multilingual and task-specific experts is quite striking. Does this separation remain consistent across different expert selection thresholds (τ)? If the threshold were lowered, could there be experts that participate in both language and task representations, and what might that mean for the design or effectiveness of your intervention?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow, with effective visualizations and well-designed, information-theory informed probes to analyze routing patterns in multilingual data. The analysis  using an entropy-normalized JS divergence metric to study routing divergence, complemented by routing entropy and consistency analyses is good to me.\n\n2.The work includes comparisons across model scales and multiple languages, enhancing the generality of the findings.\n\n3. . The figures, especially Figure 1 and Figure 2, are excellent visualizations that clearly communicate the central hypothesis of the U-shaped routing divergence and its correlation with model performance.\n\n4. The finding of complete modularity between task-specialized and language-specialized experts (Finding 5) is particularly strong and clean.", "weaknesses": "1. The proposed method improves performance by only 1–2%, which is not practically meaningful. The analysis and intervention seem too complex for such minimal payoff, which makes it hard to justify the paper’s overall significance.\n\n2. The proposed “steering” approach feels like a post-hoc, brute-force fix. Forcing a model to use certain experts is more of a hack than a genuine solution. The method shows that steering can slightly help, but not that it’s a meaningful or scalable way forward.\n\n3. The observation of middle layers of LLMs form a shared language space has already been well established for dense models. Showing that MoEs follow the same pattern is interesting but not surprising. The paper essentially confirms existing ideas rather than offering something genuinely new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0UpFkQfkIs", "reviewer": "Reviewer_JWgw", "summary": "This work conducts an in-depth study on the routing pattern  of MoE models on multi-lingual data. Authors conclude five findings via visualization and one more finding about how to leverage the observations to further improve the model.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) The writing is very clear. It is great to highlight the findings in this paper as it is indeed a bit easy to get lost when reading the detailed analysis and the description of experimental setups.\n2) The finding 6 is a good evidence helping support the correctness of the findings via visualisation. It is also interesting that simply changing the routing decision by aligning with english can improve the performance. \n3) Consider MoE is becoming a default choice of most LLMs. It indeed motivates the community to rethink the strength and weakness of using such a sparse model. For the long term purpose, as the compute is getting cheaper and cheaper, we might roll back to dense for better and more robust universal representations.", "weaknesses": "1) It would be helpful to provide more insights about how these findings are connecting to each other, and maybe draw a figure to explain this in the final draft.\n2) Minor: To my best knowledge, this work (https://arxiv.org/abs/2402.01739) should be one of the earliest work taking a closer look at this problem. I understand that the model has been a bit too weak and some designs have been outdated, but it would be helpful to discuss this work as a reference properly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lBDBM7pUDB", "reviewer": "Reviewer_pESw", "summary": "This paper studies the behavior of different languages ​​in the routing mechanism of a mixture-of-experts (MoE) model in multilingual scenarios and proposes a method to improve multilingual performance by intervening in routing. This paper proposes a soft intervention approach, which adjusts expert scores before routing, and a hard intervention approach, which forcibly activates certain experts. Experimental results demonstrate that the proposed method effectively improves the model's performance in other languages.", "questions": "1. The authors do not explain why closer approximation to English leads to improved performance. Can similar effects be observed by making a low-resource language closer to a high-resource language?\n\n2. There are no qualitative experiments demonstrating the inference process after intervention, or whether using an English expert leads to changes in the output language during inference.\n\n3. The authors' proposed method involves many hyperparameters. To what extent do the choices of these hyperparameters affect the results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors conduct an in-depth exploration of the behavior of different languages ​​in the MoE's large language model and summarize four findings.\n\n2. There is a close connection between the proposed method and the findings.\n\n3. The proposed method can improve the performance of the model in other languages ​​without further training.", "weaknesses": "1. Lack of further explanation for how the proposed method work.\n\n2. Experiments lacking qualitative analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)