Multilingual Routing in Mixture-of-Experts
Authors: name: Lucas Bandarkar, name: Chenyuan Yang, name: Mohsen Fayyaz, name: Junlin Hu, name: Nanyun Peng
Organization: University of California, Los Angeles

Mixture-of-Experts (MoE) architectures have become the key to scaling modern
LLMs, yet little is understood about how their sparse routing dynamics respond
to multilingual data. In this work, we analyze expert routing patterns using
parallel multilingual datasets and present highly interpretable layer-wise
phenomena. We find that MoE models route tokens in language-specific ways in
the early and late decoder layers but exhibit significant cross-lingual routing
alignment in middle layers, mirroring parameter-sharing trends observed in
dense LLMs. In particular, we reveal a clear, strong correlation between a
model's performance in a given language and how similarly its tokens are routed
to English in these layers. Extending beyond correlation, we explore
inference-time interventions that induce higher cross-lingual routing
alignment. We introduce a method that steers the router by promoting
middle-layer task experts frequently activated in English, and it successfully
increases multilingual performance. These 1-2% gains are remarkably consistent
across two evaluation tasks, three models, and 15+ languages, especially given
that these simple interventions override routers of extensively trained,
state-of-the-art LLMs. In comparison, interventions outside of the middle
layers or targeting multilingual-specialized experts only yield performance
degradation. Altogether, we present numerous findings that explain how MoEs
process non-English text and demonstrate that generalization is limited by the
model's ability to leverage language-universal experts in all languages.