Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of
  Language Family Experts
Authors: name: Guorui Zheng; name: Xidong Wang; name: Juhao Liang; name: Nuo Chen; name: Yuping Zheng; name: Benyou Wang
Published: 2024-10-14 15:31:54

Abstract: Adapting medical Large Language Models to local languages can reduce barriers
to accessing healthcare services, but data scarcity remains a significant
challenge, particularly for low-resource languages. To address this, we first
construct a high-quality medical dataset and conduct analysis to ensure its
quality. In order to leverage the generalization capability of multilingual
LLMs to efficiently scale to more resource-constrained languages, we explore
the internal information flow of LLMs from a multilingual perspective using
Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE
routing method that employs language-specific experts and cross-lingual
routing. Inspired by circuit theory, our routing analysis revealed a Spread Out
in the End information flow mechanism: while earlier layers concentrate
cross-lingual information flow, the later layers exhibit language-specific
divergence. This insight directly led to the development of the Post-MoE
architecture, which applies sparse routing only in the later layers while
maintaining dense others. Experimental results demonstrate that this approach
enhances the generalization of multilingual models to other languages while
preserving interpretability. Finally, to efficiently scale the model to 50
languages, we introduce the concept of language family experts, drawing on
linguistic priors, which enables scaling the number of languages without adding
additional parameters.