Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts

## Abstract

Abstract Adapting medical Large Language Models to local languages can reduce barriers to accessing healthcare services, but data scarcity remains a significant challenge, particularly for low-resource languages. To address this, we first construct a high-quality medical dataset and conduct analysis to ensure its quality. In order to leverage the generalization capability of multilingual LLMs to efficiently scale to more resource-constrained languages, we explore the internal information flow of LLMs from a multilingual perspective using Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE routing method that employs language-specific experts and cross-lingual routing. Inspired by circuit theory, our routing analysis revealed a “Spread Out in the End“ information flow mechanism: while earlier layers concentrate cross-lingual information flow, the later layers exhibit language-specific divergence. This insight directly led to the development of the Post-MoE architecture, which applies sparse routing only in the later layers while maintaining dense others. Experimental results demonstrate that this approach enhances the generalization of multilingual models to other languages while preserving interpretability. Finally, to efficiently scale the model to 50 languages, we introduce the concept of language family experts, drawing on linguistic priors, which enables scaling the number of languages without adding additional parameters.

## 1 Introduction

The development of medical large language models (LLMs) holds great promise in addressing global healthcare inequalities (Yan et al., ). By democratizing access to expert knowledge, LLMs can help mitigate disparities in resource availability within healthcare systems worldwide (Tariq et al., ). A critical aspect of ensuring this accessibility is the inclusion of local languages, which can significantly reduce barriers to adoption and foster more equitable healthcare services (Dai et al., ; Permanyer et al., ).

However, despite the potential of multilingual LLMs in healthcare, significant challenges persist. A major obstacle is the scarcity of medical data in many languages, limiting model development for underrepresented populations. While some research has made strides in addressing this issue, these efforts often focus on only a few dominant languages, typically fewer than six (Wang et al., ; Qiu et al., ). To address this gap, we have expanded the dataset to include 12 major languages, thereby improving population representation and rigorously evaluating the dataset’s quality and scalability.

Expanding the coverage from 12 major languages to include low-resource languages presents greater challenges due to data scarcity. Addressing this issue requires analyzing the internal information flow of large language models from a multilingual perspective to develop a generalizable approach. While some studies employ neuron analysis to investigate this (Tang et al., ), they typically focus on fewer than seven languages, and the complexity of neuron decomposition limits the exploration of relationships between languages and the scalability of applications. To enhance the understanding of these mechanisms, we leverage the modularity of Mixture of Experts models and introduce Hybrid-$k$ routing. This method not only ensures the activation of language-specific experts but also achieves performance on par with the vanilla top-$k$ approach, balancing interpretability and performance.

Inspired by the theory of Circuits (Olah et al., ), we regard experts as nodes, and consider the directed acyclic graph formed by the information transmission of each token from shallow to deep layers as ”Circuits”. Through observation and analysis, we identify the “Spread Out in the End” mechanism in the information flow circuits, where cross-lingual integration occurs in the early layers, while language-specific differentiation happens in the later layers. This insight suggests that employing the Post-MoE architecture, where the MoE structure is applied only in the later layers. Experiments demonstrates that the performance of 12 major languages remained stable, while low-resource languages improved even without additional training.

Building on above foundation, we further explore an efficient approach to extend the model’s multilingual medical capabilities to 50 languages. Leveraging linguistic priors, we group languages into language families, reducing the number of expert layers required for multilingual expansion from 50 to 7. Through extensive experiments with models of 0.5B, 1.5B, and 7B parameters, we demonstrate the scalability of this method and introduce the Apollo-MoE series(^1^11Named after Apollo, the Greek god of medicine and light). The series demonstrates significant potential for expansion to more languages across 50 languages, allowing for the continued increase in the number of languages without the need for additional parameters while maintaining multilingual generalization.

The main contributions are as follows:
1) We construct a high-quality medical dataset encompassing 12 major languages, quality of which is validated by experiments.
2) We propose a new circuits-based paradigm for interpreting routing in a multilingual context. Through circuit analysis, we identify the “Spread Out in the End” mechanism.
3) By introducing language family experts, we efficiently extend medical LLMs to 50 languages, demonstrating its potential for scaling to more languages.

## 2 A Preliminary Scaling to 12 Languages

We commence our work with data collection. Sec. 2.1 will outline the philosophy and pipeline for collecting and processing data, while Sec. 2.2 will address quality checks and ablation studies related to data construction.

### 2.1 Data Collection and Procession

Figure: Figure 1: Taxonomy and Token statistics of Training Dataset.
Refer to caption: x1.png

Data Collection  According to the key sources from which doctors and medical students acquire knowledge, we categorize these into seven valuable sources: Books, Papers, Encyclopedias, Doctor-Patient Dialogues, Exams, Websites, and Practical Guidelines, ensuring data quality from the outset. Additionally, we include general instruction tuning data to maintain foundational skills, as well as Math and Code data to enhance reasoning capabilities. Utilizing these sources, we gather high-quality data under open source licenses from the Internet across 12 languages, selected based on population coverage. For specific collection sources of the dataset, please refer to App. A.1.

Data Processing  Inspired by recent work on data construction during the Instruction Tuning phase (Cheng et al., ; Yue et al., ; Chen et al., ), we employ ChatGPT(^2^22gpt-3.5-turbo-16k-0613) to transform text into question-answer pairs, enhancing the quality of our dataset. For data leakage checks, we adhere to the detection strategy outlined in Med-PaLM2 (Singhal et al., ). Specifically, if an entire question or at least 64 consecutive characters overlapped with any data item, we classify that data item as a leakage instance. In our examination of the exam data sources, we began with 621,291 exercises, from which we removed 3,479, yielding a screening rate of 0.56%. For other data sources, the filtering ratio was less than 0.01%. Ultimately, we compile a multilingual medical training set containing 2.5 billion tokens, with statistical data for various languages and sources illustrated in Fig.1. More detailed information on data processing and relevant prompts can be found in App. A.2.

Evaluation Setup  To ensure the validity of the benchmark, we utilize the multilingual medical benchmark that is publicly available and peer-reviewed. For evaluating languages with limited resources, we follow the multilingual evaluation methodology of Llama3 (Dubey et al., ), employing Google Translate to translate the questions and answers of MMLU (Hendrycks et al., ). We employed a random selection of 3-shot queries to pose questions to the model, followed by answer extraction and evaluation of the model’s responses. For detailed information regarding the composition of the evaluation set and the evaluation sample used, please refer to the App. A.3.

### 2.2 Experimental Results

While we ensure data quality during source evaluation, we also perform a statistical analysis of the dataset’s quality and composition. First, we conduct monolingual training, which involves training the model solely on data from a specific language to assess dataset’s quality in that language. Next, we perform an ablation study focused on the function of code and math to determine the necessity of including this data. Multilingual training, which incorporates all available data using a random sampler, serves as our default reference. We select Gemma-2b (Mesnard et al., ) as our base model due to its moderate number of parameters. The evaluation setting and training setting used in this section are consistent with other experiments, see App. B.

**Table 1: Monolingual ( ) refers to the accuracy of models trained on single language data and evaluated on the respective language evaluation set. Multilingual ( ) indicates the accuracy of models trained and evaluated on all languages’ datasets. The -Math&Code results reflect performance when training on all language data ( ) excluding math and code data. Avg. denotes the average score across languages.**
| Model | #params | Avg. | Ar | De | En | Es | Fr | Hi | It | Ja | Ko | Pt | Ru | Zh |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Gemma | 2B | 28.1 | 18.2 | 27.6 | 37.6 | 31.8 | 23.1 | 25.7 | 26.1 | 21.2 | 25.2 | 23.1 | 49.6 | 28.0 |
| +Monolingual | 2B | - | 27.8 | 34.9 | 52.1 | 39.2 | 27.7 | 27.6 | 28.6 | 23.4 | 29.9 | 28.3 | 54.7 | 60.6 |
| +Multilingual | 2B | 48.6 | 43.7 | 50.7 | 57.8 | 48.1 | 44.5 | 40.7 | 45.2 | 43.0 | 45.1 | 40.6 | 63.7 | 60.6 |
| -Math&Code | 2B | 42.7 | 36.1 | 39.4 | 51.2 | 43.7 | 39.3 | 35.4 | 42.0 | 35.4 | 40.5 | 32.1 | 57.0 | 60.0 |

Results  As shown in Tab. 1, our quality checking indicates that models trained separately with language-specific data exhibit improved performance on corresponding tests, further confirming the high quality of the dataset in each language. Regarding the inclusion of math and code data, the model experiences an average performance loss of 5.9% when trained without this data compared to training with the full dataset, underscoring the significance of math and code for model performance. This enhancement may be attributed to the ability of math and code data to strengthen the model’s reasoning capabilities. Additionally, numerals and coding language, as common elements across languages, likely serve as anchor points in multilingual training, facilitating mutual alignment of language distributions. We also utilize the dataset to train models of various architectures and sizes to further validate its effectiveness, with related details and results presented in App. A.4.

## 3 Scaling with MoE and its Routing Analysis

This section leverages Mixture of Experts (MoE) to scale medical LLMs for better efficiency and extensibility. Sec. 3.1 introduces a new routing method and its experimental validation. Sec. 3.2 delves into a detailed analysis of the routing mechanisms using circuits; where we observe the “Spread Out in the End” phenomenon: expert routing paths are shared among languages in early layers and diverge in later layers. Inspired by the phenomenon, Sec. 3.3 presents a MoE variant called “Post-MoE”, which restricts routing to the later layers.

### 3.1 A Language-specific Hybrid Routing

While Sec. 2 utilizes a dense model to integrate multilingual data and extend the medical model to 12 languages, this approach presents efficiency limitations. To address these challenges, we adopt a sparse Mixture of Experts (MoE) model for a better balance between effectiveness and efficiency. The modular and functional nature of the Experts in the MoE framework is particularly advantageous in
further scalability, which benefit enabling the effective expansion to more languages (especially for minor languages in Sec. 4.

#### 3.1.1 The Philosophy of Hybrid-k

Figure: Figure 2: Hybrid routing ensures that the experts corresponding to the input token language are activated. As illustrated, if the weight of the language-specific expert do not rank among the top two, it will replace the expert with lower weights; otherwise, no changes will be made.
Refer to caption: x2.png

We propose new MoE consists of language-specific experts and hybrid routing, enhancing both language-specific expertise and transfer of general medical knowledge across languages.

Language-specific Experts Inspired by Li et al. (); Pfeiffer et al. (); Kwon & Chung (), we leverage the MoE structure to modularize the language-specific parameters in the medical domain. Specifically, we design language-specific experts to more effectively handle language-dependent knowledge and inputs. However, this has notable drawbacks: the knowledge encapsulated within each expert tends to be isolated, which can impede the learning of general medical knowledge.

Hybrid Routing To address this limitation and enhance general knowledge acquisition across languages, we propose cross-lingual routing within the MoE. This allows routing to go beyond language-specific experts, enabling knowledge to propagate across languages. As shown in Fig. 2, the result is a hybrid mechanism, where routing can target both language-specific Experts and dynamically route to other language experts; the latter is related to the input text itself.

In Hybrid-$k$, tokens can be routed not only to language-specific experts but also to cross-lingual experts. The rationale for cross-lingual routing is to view text as a tool for thought, capable of being expressed through various languages. Given the exceptional multilingual processing abilities of LLMs, it can be inferred that they adeptly switch between and intertwine multiple languages during text comprehension.

#### 3.1.2 Experimental Comparisons between Routing Strategies

To validate the effectiveness of our method, we compared it with different routing strategies, specifically vanilla Top-$k$ and Language-Specific (Lang-Spec.) routing. Lang-Spec. routing refers to selecting the corresponding expert based on the input language type, while keeping a shared expert constantly activated to enhance performance and align the model’s inference parameters.

Experiment Settings Considering its broad parameter base and impressive multilingual capabilities, we selected the Qwen2 series for our experiments. Specifically, we conducted experiments using the Qwen2-0.5B model, based on the training and evaluation datasets described in Sec. 2. For token language classification, tokens are uniformly classified based on their source per document.
To construct the evaluation set for minor languages, we evenly select 38 languages based on their geographical distribution. Similar to Sec. 2 , we use Google Translate to translate the questions and answers from the medical-clinical section of the MMLU dataset, types of minor languages are detailed in the App. A.5.
For Lang-Spec. routing, we use 12 experts plus one shared expert. For Hybrid-$k$ and Top-$k$ routing, 12 experts are used per layer. The activation count is fixed at two experts across all configurations ($k=2$) if not specified. Additional training settings are provided in App. B.

**Table 2: Comparison between Dense models and MoE models with various routing strategies.**
| Method | Avg. Accuracy | Ar | De | En | Es | Fr | Hi | It | Ja | Ko | Pt | Ru | Zh |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Major | Minor |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Dense Models before and after Training |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen-0.5B | 29.7 | 31.5 | 27.3 | 27.4 | 39.3 | 32.9 | 21.3 | 25.7 | 20.7 | 24.0 | 19.2 | 26.3 | 46.9 | 45.4 |
| after training | 37.8 | 24.6 | 34.8 | 33.7 | 46.7 | 40.5 | 33.0 | 31.0 | 31.5 | 31.5 | 35.5 | 30.8 | 53.8 | 51.0 |
| MoE Models Trained with Different Routing Strategies |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Lang-Spec. | 30.9 | 26.1 | 28.8 | 28.6 | 39.1 | 33.3 | 16.5 | 24.5 | 21.4 | 27.8 | 31.0 | 52.3 | 42.8 | 53.0 |
| Top-$k$ | 39.7 | 29.9 | 34.5 | 36.9 | 43.7 | 38.9 | 39.9 | 32.2 | 37.7 | 30.5 | 35.9 | 34.8 | 58.2 | 53.2 |
| Hybrid-$k$ | 40.0 | 32.0 | 35.1 | 37.2 | 44.1 | 40.8 | 40.7 | 28.7 | 38.9 | 32.4 | 36.2 | 34.3 | 58.8 | 53.6 |

Results
Tab. 2 shows that the base model in major languages is improved significantly after fine-tuning, but its performance in minor languages declines substantially. This indicates that partial language fine-tuning of dense models notably impacts their generalization capabilities to other languages. The Language-Specific routing of MoE provides interpretability but results in poor multilingual performance. In contrast, the Top-$k$ MoE model demonstrates superior multilingual capability and generalization after fine-tuning compared to the trained base model, highlighting the effectiveness of MoE models over dense models. Moreover, Hybrid-$k$ routing exhibits a clear advantages than Top-$k$ in nearly all languages, especially in minor languages; this evidences its superior generalization.

### 3.2 Interpretable Routing Analysis: Information Flow Circuit

To better interpret the routing patterns in multilingual context, we propose to formulate the routing pattern across layers as circuit in Sec. 3.2.1 and conduct some visualized study in Sec. 3.2.2.

#### 3.2.1 Formulation of Information Flow in Routing

During the routing process, each token is routed to both its language-specific expert and other language experts, enabling cross-lingual routing. This section aims to analyze the cross-lingual routing patterns for each language. Specifically, we seek to understand how tokens in a given language benefit from other language experts as they traverse from lower to higher layers. In other words, beyond its own language, the key question is: How does input in each language leverage other language experts across different layers? As mentioned in Sec. 3.1.1, the involvement of intermediate language experts may provide insights into the languages the model utilizes for internal thinking and reasoning.

To investigate the process, we propose a circuit-based formulation as below:

Definition 1. Information Flow Circuit
The sequence of Experts that each token passes from shallow to deeper abstractions forms a directed acyclic graph (DAG), which we refer to as a ‘circuit’.

Figure: Figure 3: Visualization for routing patterns. Right: Hybrid-$k$ routing distribution. The x-axis represents language experts, with values indicating the proportion of tokens allocated to each expert. AVG denotes the aggregated routing distribution across 12 languages. Left: Visualization of AVG routing distribution from the perspective of Information Flow Circuits. We retained expert nodes with a token ratio of 0.5.
Refer to caption: extracted/5921635/pic/circuit.png

To examine expert routing within the Hybrid-$k$ routing, we conduct an experiment to track token routing. We extracted varying amounts of data from 12 languages (details on data quantity and format are provided in App. D) to obtain approximately 80,000 tokens per language. These 12 single-language datasets were used as probing sets to record expert routing at the token level.

#### 3.2.2 Results

The visualization of the routing pattern is shown in Fig. 3. From the perspective of single-language routing, the routing results for Japanese and Spanish in the figure (routing distributions for other languages are detailed in the App.E) indicate that Chinese has a significant influence on Japanese, while Portuguese and English are also integrated into the information flow of Spanish. This phenomenon aligns well with linguistic priors: the linguistic development of Japanese has been influenced by Chinese, while Spanish, Portuguese, and English, all belonging to the Romance language family, have historically exerted mutual influence. From the perspective of cross-linguistic routing for all 12 languages, through the utilization of routing distribution analysis and Information Flow Circuits visualization, we observe that the information flow circuits exhibit cross-linguistic concentration in the early layers, while differentiation based on language occurs in the later layers. We refer to this phenomenon as “Spread Out in the End”:

Phenomenon 1. In the early layers, the model exhibits shared routing patterns across multiple languages. However, in the later layers, the model diverges, with tokens being routed to language-specific experts, allowing late routing to specialize in its respective language.

### 3.3 A MoE variant inspired by the ‘Spread Out in the End’ phenomenon: Post-MoE

Figure: (a) Post-MoE from Qwen2-0.5B in Last N Layers
Refer to caption: extracted/5921635/pic/Post-MoE-0.5B.png

Inspired by the phenomenon of “Spread Out in the End,” we propose the Post-MoE architecture, which applies the Mixture of Experts (MoE) structure only in the final layers. Using this architecture, we further validate the observed phenomenon and investigate the impact of the number of MoE layers in the model’s final layer on its performance.

Experiment Settings To validate the effectiveness of the multilingual mechanism and the Post-MoE architecture, we use Qwen2-0.5B-MoE and Qwen2-1.5B-MoE, along with the original base model as a baseline. Qwen2-0.5B-MoE and Qwen2-1.5B-MoE are standard MoE architectures trained with Hybrid routing. To investigate the impact of the number of MoE layers in the final layer of the model on performance, we extend the MoE architecture in the last 1, 2, 3, and 4 layers using the hybrid routing method. We also evaluate the multilingual generalization capability of this architecture using the assessment set of 38 minor languages mentioned in Sec. 4.

Results  As shown in Fig. 4, the architecture with MoE extended in the last two layers achieves the best performance, balancing both accuracy and multilingual generalization. The experimental results align closely with the routing pattern visualization in Fig. 3, further validating the “Spread Out in the End” phenomenon. In the next section, we will further apply this method to 50 languages to fully leverage the advantages and scalability of this architecture.

## 4 Further Scaling to 50 Languages

To further demonstrate the multilingual capabilities of the Post-MoE architecture, we selected 38 minor languages, expanding the language variety to 50 (shown in Tab. 3).

**Table 3: Classification of Languages with Color Coding Based on Characteristics.**
| Language Family | Languages |
| --- | --- |
| Sino-Tibetan | Chinese (Zh) |
| Altaic | Korean (Ko), Japanese (Ja), Mongolian (Ne) |
| Australasian | Thai (Th), Vietnamese (Vi), Laotian (Lo) |
| Austronesian | Malagasy (Mg), Cebuano (Ceb), Sundanese (Su), Ilokano (Ilo), Dogue (Doi) |
| Indo-European | English (En), German (De), Portuguese (Pt), Spanish (Es), French (Fr), Russian (Ru) |
|  | Italian (It), Croatian (Hr), Galician (Gl), Czech (Cs), Corsican (Co), Latin (La), |
|  | Ukrainian (Uk), Bosnian (Bs), Bulgarian (Bg), Esperanto (Eo), Maithili (Mai), |
|  | Albanian (Sq), Danish (Da), Sanskrit (Sa), Norwegian (No), Guarani (Gn), |
|  | Serbian (Sr), Slovak (Sk), Scottish Gaelic (Gd), Luxembourgish (Lb), Hindi (Hi) |
| Afro-Asian | Arabic (Ar), Kurdish (Sorani) (Ckb), Maltese (Mt), Hebrew (He) |
| Kongolese | Lingala (Ln), Bambara (Bm), Swahili (Sw), Sepeti (Nso), Igbo (Ig), |
|  | Kinyarwanda (Rw), Hausa (Ha) |

### 4.1 Mixture of Language Family Experts

Following the Hybrid-$k$ which adopts language-specifc experts, training LLMs with $n$ languages would require $n$ language-specific experts. This expansion strategy would lead to an substantial growth in model parameters, making the application impractical. To address this challenge, we propose adapting Post-MoE by introducing the Mixture of Language Family Experts, which groups the 50 languages into 7 established linguistic families, as detailed in Tab. 3. This approach named ‘Apollo-MoE’ ensures that scaling to additional languages does not necessitate a corresponding increase in parameters. Each language family employs hybrid routing tailored to its linguistic characteristics, enabling more efficient training. This method might facilitate scalable and robust multilingual performance across a wide range of languages.

### 4.2 Experiments

**Table 4: Main Results on 50 languages comparing to existing LLMs**
| Model | #params | Average Acc. | Ar | De | En | Es | Fr | Hi | It | Ja | Ko | Pt | Ru | Zh |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Major | Minor |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Closed-source Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gemini 1.5 Pro | - | 74.6 | 73.6 | 74.9 | 80.4 | 63.2 | 81.9 | 87.5 | 75.4 | 69.1 | 68.9 | 78.6 | 79.9 | 64.8 | 70.0 |
| GPT-4 Turbo | - | 79.4 | 73.3 | 79.8 | 84.7 | 68.5 | 85.8 | 89.7 | 72.8 | 80.9 | 77.5 | 79.5 | 87.0 | 78.1 | 68.9 |
| Claude 3 Opus | - | 81.9 | 76.5 | 78.8 | 87.2 | 69.8 | 87.9 | 90.7 | 78.6 | 84.6 | 82.1 | 85.2 | 86.7 | 80.1 | 70.9 |
| GPT-4o | - | 85.7 | 81.1 | 80.9 | 90.5 | 70.5 | 90.7 | 93.5 | 86.9 | 85.1 | 89.5 | 90.2 | 90.8 | 78.5 | 80.9 |
| GPT-4o mini | - | 77.6 | 69.8 | 74.3 | 80.4 | 70.2 | 82.3 | 84.5 | 75.4 | 77.6 | 76.2 | 77.1 | 82.3 | 80.9 | 67.5 |
| Open-source Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| JSL-MedPhi2 | 2.7B | 29.0 | 30.7 | 25.0 | 30.8 | 42.1 | 34.6 | 30.6 | 13.2 | 26.6 | 17.9 | 21.3 | 29.7 | 48.8 | 26.8 |
| MMed-Llama-3 | 8B | 40.2 | 36.5 | 29.5 | 40.3 | 54.7 | 63.6 | 62.0 | 38.8 | 20.7 | 47.3 | 20.3 | 25.9 | 62.5 | 16.7 |
| OpenBioLLM | 8B | 46.1 | 34.7 | 27.5 | 58.1 | 49.6 | 59.2 | 53.3 | 44.4 | 41.5 | 39.1 | 46.7 | 27.0 | 64.8 | 41.8 |
| Llama3 | 8B | 49.3 | 33.3 | 33.5 | 55.9 | 48.4 | 60.5 | 58.4 | 48.0 | 39.9 | 38.9 | 49.5 | 51.5 | 63.3 | 43.6 |
| Our Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Apollo-MoE | 0.7B | 40.5 | 34.6 | 36.3 | 38.2 | 45.4 | 39.8 | 38.4 | 33.1 | 39.9 | 26.9 | 35.2 | 37.3 | 64.1 | 51.3 |
| Apollo-MoE | 2.0B | 54.8 | 44.9 | 47.2 | 53.8 | 56.5 | 52.5 | 53.3 | 39.5 | 54.4 | 45.7 | 49.5 | 57.3 | 69.1 | 66.8 |
| Apollo-MoE | 10.1B | 69.9 | 58.3 | 58.3 | 73.5 | 73.1 | 69.4 | 72.4 | 56.9 | 71.9 | 62.4 | 68.4 | 73.8 | 74.2 | 84.1 |

Experiment Settings  Given the extremely limited data for these rare languages, we extracted 2,000 samples from English data and used Google Translate to create training corpora for each language. The clinical-knowledge section of MMLU was translated to serve as the corresponding evaluation set.
We trained the Post-MoE model, named Apollo-MoE, using Qwen2-0.5B, 1.5B, and 7B as base models with a Mixture of Language Family Experts. We evaluated Apollo-MoE on benchmarks for major and minor languages and compared it with open-source and high-performing open-source medical models.

Results
Tab. 4 demonstrate that our Apollo-MoE models outperform other models of similar sizes in both major and minor languages. The 2B model achieves 54.8 in major and 44.9 in minor languages, surpassing open-source models with 8B parameters. The 10B model leads all open-source models, achieving 69.9 in major and 58.3 in minor languages. Notably, Apollo-MoE excels in major languages like English, French, and Spanish, with particularly strong performance in French, and outperforms other models significantly in low-resource languages like Arabic and Hindi.

Ablation Study
Tab. 5 provides a detailed comparison of routing performance across 50 languages, with two experts routed among 7 linguistic family experts in the Post-MoE model.
It shows that routing strategies consistently outperform Dense models, with Hybrid-$k$ routing achieving an average major language accuracy of 54.8 for Qwen2-1.5B, compared to 52.2 for Dense.
Additionally, Hybrid-$k$ routing shows a clear advantage over Top-$k$ routing, particularly in minor languages, where the Qwen2-7B model achieves an average accuracy of 58.3 with Hybrid-$k$, compared to 56.7 with Top-$k$.

**Table 5: Ablation Study on 50 Languages.**
| Languages | Average acc. | Ha | Sr | La | Gn | Doi | Da | Ln | Ceb | Mai | Mg | Ilo |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Major | Minor |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2-0.5B | 29.7 | 31.5 | 31.1 | 30.7 | 36.4 | 28.4 | 27.7 | 33.3 | 29.9 | 34.5 | 27.3 | 34.8 | 30.7 |
| Dense | 39.2 | 33.2 | 33.7 | 25.0 | 30.7 | 33.3 | 32.6 | 36.0 | 34.5 | 36.7 | 31.8 | 34.1 | 35.2 |
| Top-$k$ routing | 39.0 | 33.4 | 26.9 | 17.8 | 37.5 | 34.1 | 29.9 | 36.7 | 33.7 | 32.2 | 28.8 | 32.2 | 32.6 |
| Hybrid-$k$ routing | 40.5 | 34.6 | 31.1 | 25.4 | 38.3 | 34.8 | 36.0 | 39.8 | 41.3 | 40.9 | 33.7 | 37.9 | 39.8 |
| Qwen2-1.5B | 42.9 | 38.4 | 36.0 | 44.7 | 39.0 | 37.5 | 31.8 | 45.1 | 34.8 | 42.4 | 34.1 | 39.0 | 35.6 |
| Dense | 52.2 | 43.7 | 36.7 | 30.3 | 51.1 | 47.0 | 37.1 | 51.1 | 44.3 | 48.9 | 39.8 | 40.5 | 45.8 |
| Top-$k$ routing | 53.6 | 42.6 | 39.8 | 24.6 | 47.7 | 44.7 | 36.0 | 50.4 | 34.5 | 48.5 | 35.6 | 41.7 | 43.9 |
| Hybrid-$k$ routing | 54.8 | 44.9 | 39.0 | 33.7 | 48.5 | 45.8 | 41.3 | 55.3 | 43.9 | 50.8 | 42.8 | 41.7 | 46.2 |
| Qwen2-7B | 55.2 | 49.2 | 34.1 | 57.6 | 52.3 | 43.2 | 40.9 | 63.3 | 38.3 | 58.3 | 48.5 | 37.1 | 45.5 |
| Dense | 69.0 | 55.7 | 37.9 | 31.1 | 60.6 | 54.9 | 55.7 | 68.6 | 47.3 | 64.4 | 55.7 | 44.7 | 64.4 |
| Top-$k$ Routing | 68.6 | 56.7 | 37.1 | 39.4 | 60.6 | 56.1 | 50.8 | 67.8 | 53.4 | 67.0 | 59.8 | 46.2 | 61.0 |
| Hybrid-$k$ Routing | 69.9 | 58.3 | 48.5 | 42.0 | 62.9 | 61.0 | 54.5 | 71.2 | 53.0 | 68.6 | 58.3 | 50.8 | 64.4 |

Figure: Figure 5: Data Scale ACC.
Refer to caption: extracted/5921635/pic/Data_scaling_law.png

On the Data Scale
To investigate the sensitivity to data scale, we trained Post-MoE model with Qwen2-0.5B base using a combination of major languages along with 10%, 30%, 50%, and 100% of the data from minor languages, corresponding to 200, 600, 1,000, and 2,000 data examples per language, respectively. As shown in Fig. 5, the model’s performance across various languages improves with increasing data but eventually plateaus. This indicates that the proposed method is not heavily data-dependent, reaching saturation with as few as 2,000 data examples; this is especially beneficial for low-resources languages.

## 5 Related Work

Mixture-of-Experts and Sparse Upcycling
Mixture-of-Experts (MoE) models differ from traditional dense models by activating only a subset of parameters, or ”experts,” for each input token. This selective activation reduces computational costs while preserving model capacity. First introduced by Shazeer et al. (), MoEs have since evolved in models like GShard (Lepikhin et al., ), Switch Transformers (Fedus et al., ), and Mixtral (Jiang et al., ), all of which have shown enhanced performance in large-scale language tasks. Recent advancements, such as Sparse Upcycling (Komatsuzaki et al., ), have efficiently initialized MoEs by replicating pre-trained dense model parameters across MoE layers, specifically duplicating feed-forward network (FFN) parameters for each expert. BTX (Sukhbaatar et al., ) extends this by upcycling multiple specialized models, further refining MoE efficiency and specialization.

Multilingual Medical LLMs
Recent research has increasingly focused on expanding multilingual medical data and developing models tailored to linguistically diverse populations. A LLM integrated with machine translation has been proposed to address the scarcity of multilingual medical datasets (Gangavarapu, ).
Furthermore, multilingual medical corpora and benchmarks across six languages have been developed (Qiu et al., ; Wang et al., ). Our work builds on these efforts by investigating multilingual patterns and expanding it to 50 languages.

Multilingual Capability Enhancement Research on Multilingual Models has primarily aimed to enhance multilingual capabilities and understand their mechanisms. Efforts have improved performance through translation (Liang et al., ; Huang et al., ) and cross-lingual alignment (Salesky et al., ). Techniques like cross-lingual transfer (Kim et al., ; Lin et al., ) and continuous training in specific languages have further advanced LLMs (Cui et al., ), while training from scratch shows potential (Muennighoff et al., ). Despite these advancements, many methods are limited by task specificity or extensive data requirements, leaving the multilingual mechanisms of LLMs insufficiently explored. Recent models (Zhao et al., ; Nguyen et al., ) exhibit strong multilingual abilities without explicit language alignment. Tang et al. () and Zhao et al. () employ neuron analysis method (Mu & Andreas, ) to investigate the mechanism, but they typically focus on fewer than seven languages. Additionally,
the complexities of neuron disassembly further limit the applicability of these findings.

Mechanistic Interpretability Circuit analysis has emerged as a new paradigm of model analysis, providing an in-depth mechanistic interpretability of the internal information flow and hierarchical structure of models (Olah et al., ; Merullo et al., ). Additionally, the study of modularity (Meng et al., ; Wang et al., ) and sparcity (Olsson et al., ) within models offers insights into constructing specialized submodules, which helps to organize information sharing and feature isolation more effectively. Component-level detailed analysis (Olsson et al., ; Vig et al., ; Singh et al., ) delves into the contributions of attention heads, neurons, and other components to model behavior. Lastly, causal mechanism analysis (Vig et al., ; Geiger et al., ) provides a method for explaining the relationships between information flow and functionality, helping to uncover the key mechanisms within models.

## 6 Conclusion

In conclusion, this work advances the development of multilingual medical LLMs by addressing key challenges in data scarcity and model scalability. We first construct a high-quality medical dataset covering 12 major languages. Subsequently, we propose Hybrid-$k$ routing to explore multilingual training in a modular manner in MoE. Based on the Hybrid-$k$ routing, we propose a circuits-based paradigm for interpreting information flow in multilingual contexts. The circuit analysis reveale the ”Spread Out in the End” mechanism, based on which we introduced the Post-MoE architecture and demonstrate its superior multilingual capabilities. Furthermore, by introducing language family experts, we efficiently extend Post-MoE’s capabilities to 50 languages with limited data, demonstrating scalability without additional parameters. It is worth noting that the proposed method can be easily extended to support a broader range of languages, which we leave as straightforward future work. These contributions lay the groundwork for broader multilingual expansion in healthcare applications, promoting more equitable access to expert medical knowledge globally.

## Limitations

Post-MoE expansion in the last two layers has shown good results. However, our base model for Post-MoE expansion only goes up to 7B parameters. For larger models, the optimal effect might not be achieved by expanding just the last two layers; a proportional calculation of layers may be required. Nonetheless, using the last two layers already yields satisfactory results for the 7B model.
Research on MoE at the module level is still in its early stages. Although Post-MoE demonstrates superior performance across multiple languages, there is still significant room for improvement in model performance.

Future Work
The Language-Specific experts and Hybrid routing are not limited to multilingual scenarios and provides a modular approach for exploring the internal mechanisms of MoE. By aligning multiple related tasks with a single expert and using experts as modules for model training, it reduces both parameter count and computational load while maintaining good performance. The Mixture of Language Family Experts in the Post-MoE architecture offers an approach for applying the model to a variety of tasks, enabling efficient expansion to more task scenarios.

## Appendix A Details of datasets, benchmark

### A.1 Dataset Collection

**Table 6: The detailed sources of training dataset.**
| Language | Source |
| --- | --- |
| Books |  |
| English | Pile Dataset (Gao et al., ) |
| Chinese | MedQA (Jin et al., ) |
| Russian | ru-medical-textbooks [[Link]](https://huggingface.co/datasets/anechaev/ru_med_history) |
| German | de-books [[Link]](https://doi.org/10.1007/BF02897758) |
| Korean | ko-books [[Link]](https://huggingface.co/datasets/maywell/korean_textbooks) |
| Papers |  |
| English | PubMed (Roberts, ) |
| Chinese | Paper from Chinese Medical Association [[Link]](https://www.yiigle.com/index) |
| French | MORFITT (Labrak et al., ), CLEAR (Grabar & Cardon, ) |
| Spanish | Mesinesp (Gasco et al., ) |
| Russian | ru-medical-paper [[Link]](https://www.kaggle.com/datasets/kwyrob/medsi-articles/data) |
| German | Germany part of Multilingual Medical Corpora [[Link]](https://zenodo.org/records/3463379) |
| Encyclopedias |  |
| ⟨Multiple⟩ | ⟨English, Russian, Hindi, Arabic, German, Italian, Korean, Japanese⟩wiki [[Link]](https://huggingface.co/datasets/legacy-datasets/wikipedia) |
| French | CLEAR (Grabar & Cardon, ) |
| Hindi | HHD corpus (Jain & Arora, ) |
| Portuguese | pt-medical-wiki [[Link]](https://huggingface.co/datasets/rhaymison/medicine-medical_meadow_wikidoc_pt) |
| Dialogues |  |
| Chinese | HuatuoGPT dataset (Zhang et al., ), Huotuo_26M (Li et al., ) |
| English | PMC-Patients (Zhao et al., ) |
| Arabic | MAQA (Abdelhay & Mohammed, ) |
| Russian | RuMedPrimeData [[Link]](https://zenodo.org/records/5765873) |
| Portuguese | askD [[Link]](https://github.com/ju-resplande/askD) |
| Italian | MedQuaAD-Italian [[Link]](https://huggingface.co/datasets/andreabac3/MedQuaAD-Italian-Fauno-Baize) |
| Korean | MedGPT-5k-ko [[Link]](https://huggingface.co/datasets/mncai/MedGPT-5k-ko), ko-medical-chat [[Link]](https://huggingface.co/datasets/squarelike/ko_medical_chat) |
| Japanese | Real-MedNLP Test Collection [[Link]](https://sociocom.naist.jp/download/real-mednlp-test-collection), ja-medial-progress-notes [[Link]](https://zenodo.org/records/4064153) |
| Exam |  |
| Chinese | CMB (Wang et al., ), CMExam (Liu et al., ), MedQA (Zhang et al., ) |
| English | MedQA, Medmcqa (Pal et al., ), Pubmedqa (Jin et al., ) |
| Spanish | HEAD-QA (Vilares & Gómez-Rodríguez, ) |
| French | Frenchmcqa (Labrak et al., ) |
| Italian | MedExpQA (Alonso et al., ) |
| Russian | RuMedBench (Blinov et al., ), BioInstructQA (Labrak et al., ) |
| Japanese | IgakuQA (Kasai et al., ) |
| Korean | KorMedMCQA (Kweon et al., ) |
| German | BioInstructQA |
| Guideline |  |
| English | NICE [[Link]](https://www.nice.org.uk/guidance), PubMed, SPOR [[Link]](https://sporevidencealliance.ca/key-activities/cpg-asset-map/cpg-database) |
| ⟨Multiple⟩ | ⟨Spanish, German, French⟩MSD-instruct [[Link]](https://huggingface.co/datasets/nuvocare/MSD_instruct) |
| Korean | Korean-guidelines-for-primary-physicians [[Link]](https://www.kaggle.com/datasets/metformin/korean-guidelines-for-primary-physicians?resource=download) |
| Web |  |
| Chinese | WUdao Dataset [[Link]](https://www.scidb.cn/en/detail?dataSetId=c6a3fe684227415a9db8e21bac4a15ab) |
| English | C4 Dataset (Raffel et al., ) |
| Spanish | CoWeSe (Carrino et al., ), medical-eval-pt [[Link]](https://huggingface.co/datasets/rhaymison/medicine-medical-eval-pt) |
| Italian | it-medical-corpus [[Link]](https://huggingface.co/datasets/HiTZ/Multilingual-Medical-Corpus), it-biomedical-dataset [[Link]](https://huggingface.co/datasets/paolo-ruggirello/biomedical-dataset) |
| German | opus-medical-de-en [[Link]](https://huggingface.co/datasets/ahazeemi/opus-medical-en-de) |
| Russian | medical-qa-ru-data [[](https://huggingface.co/datasets/blinoff/medical_qa_ru_data)Link] |
| Japanese | MedNLP-SC Social Media Corpus [[Link]](https://sociocom.naist.jp/download/mednlp-sc-sm/) |
| General |  |
| ⟨Multiple⟩ | ⟨French, Spanish, Arabic, Hindi, German, Russian, Italian, Portuguese, Japanese, Korean⟩Alpaca [[Link]](https://github.com/FreedomIntelligence/MultilingualSIFT), Sharegpt [[Link]](https://github.com/FreedomIntelligence/MultilingualSIFT) |
| Ch,En | Alpaca, Sharegpt, WizardLM Dataset |
| English | belebele benchmark (Bandarkar et al., ), ai2_arc (Clark et al., ), Capybara (Daniele & Suphavadeeprasit, ) |
| Math |  |
| ⟨Multiple⟩ | MathInstruct (Yue et al., ) |
| Code |  |
| English | Python-Alpaca [[Link]](https://huggingface.co/datasets/Vezora/Tested-22k-Python-Alpaca) |
| Chinese | Leetcode-ZH-11k [[Link]](https://huggingface.co/datasets/krisfu/awesome-llm-datasets-only-Chinese/tree/main/sft-phase-processed) |

### A.2 Dataset Procession

Adhering to the established data processing standards, we segmented sentences into chunks, filtered them for medical relevance, and reformatted them into a QA format with prompt shown in App. C.1. The processing details are as follows:

##### Books

For English books, we use medical dictionary(^3^33[https://www.nlm.nih.gov/research/umls/new_users/online_learning/LEX_001.html](https://www.nlm.nih.gov/research/umls/new_users/online_learning/LEX_001.html)) to filter the Pile Dataset and select books where medical terms account for more than 4% of the total words, resulting in 2,312 medical-related books. For Chinese books, we follow MedQA to collect medical textbooks included in the five-year and eight-year medical student training programs in mainland China, finally obtaining 90 books. For Russian books, we use Russian medical textbooks as the data source. For German medical book data, we first divided the German book data into multiple blocks with a maximum of 512 characters, then filtered the data using 2,590 highly relevant German medical terms(^4^44[https://medlineplus.gov/languages/russian.html](https://medlineplus.gov/languages/russian.html)), and finally rewrote it into QA form. Regarding Korean medical book data, we adopt the same method and Korean medical dictionary(^5^55[https://medlineplus.gov/languages/korean.html](https://medlineplus.gov/languages/korean.html)) to filter out the medical data from Korean books.

##### Papers

For English papers, we sample the public data in PubMed and obtain 878,241 medical abstracts. For Chinese papers, we also screen a total of 177,261 abstracts of papers published by the Chinese Medical Association. For French papers, we use the MORFITT dataset and the scientific article portion of the CLEAR. For the Spanish paper, we use paper abstracts open sourced by the Mesinesp. For medical literature in German and Russian, we directly divide the medical data into blocks of 512 and rewrite them into QA format for use respectively.

##### Encyclopedias

For the English Encyclopedia, we also use the English Medical Dictionary to filter out 36107 medical-related wiki pages from wiki dataset. For the French encyclopedia, we select the encyclopedia articles part of the CLEAR and filtered(^6^66[https://medlineplus.gov/languages/french.html](https://medlineplus.gov/languages/french.html)) wiki data for supplementation. For the Hindi encyclopedia, we choose the HHD corpus, which crawls descriptions of people, diseases, medical consumer products, and symptoms from Indian websites.
For Russian, Hindi, Arabic, German, Italian, and Korean, we filtered wiki data using medical dictionaries of corresponding languages(^7^77[https://medlineplus.gov/languages/languages.html](https://medlineplus.gov/languages/languages.html)). For Portuguese, we directly used the pt-medical-wiki data. As for Japanese data, we screened wiki using medical dictionaries(^8^88[https://sociocom.naist.jp/manbyo-dic-en/](https://sociocom.naist.jp/manbyo-dic-en/))(^9^99[https://sociocom.naist.jp/hyakuyaku-dic-en/](https://sociocom.naist.jp/hyakuyaku-dic-en/)) to obtain Japanese medical encyclopedia data.

##### Doctor-Patient Dialogues

For Chinese, we directly use the HuatuoGPT dataset and the simplified data set in Huatuo_26M. For English, we construct a multi-turn conversation data set based on PMC-Patients using ChatGPT with the prompt shown in Fig.8. For Arabic, we extract high-quality questions and answers with both question and answer lengths greater than 128 from the largest Arabic healthcare question and answer dataset MAQA.
For Russian, we use RuMedPrimeData from outpatient hospital patients. For Portuguese, we utilize medical Q&A data from askD. For Italian, we employ doctor-patient data from MedQuaAD-Italian. For Korean, we filter the MedGPT-5k-ko and ko-medical-chat data for medically rich doctor-patient dialogues with the dictionary. For Japanese, we rewrite QA pairs from Real-MedNLP and medical progress note to serve as Japanese doctor-patient dialogues.

##### Exams

For the Chinese exam, we collect training sets of CMB, CMExam, and MedQA. For the English exam, we collect the training sets of MedQA, Medmcqa and Pubmedqa. For the Spanish and French exam, we select the training set of HEAD-QA and Frenchmcqa  separately.
For Italian, we use the training portion defined by MedExpQA as the Italian medical exam data. For Russian, we utilize the training part from RuMedBench and the exam portion from BioiInstructQA. For Japanese, we divided IgakuQA into training and test parts at a 6:4 ratio, using them as training data for the Japanese medical exam and the Japanese medical benchmark, respectively. For Korean, we use the training part provided by KorMedMCQA. For German traning set, we employ German sections MedQA, PubMedQA, MedQA-5-options, and MedMCQA in BioInstructQA.

##### Guidelines

For English Guidelines, we select data from three sub-items of NICE, PubMed and SPOR in the clinical guidelines introduced by Meditron (Chen et al., ).
Regarding the guidelines for Spanish and German, we use data from MSD-instruct. Additionally, we incorporate the French sections as supplementary material for the French guidelines. For the Korean section, we extracted and segmented content in 1024 characters from Korean-guidelines-for-primary-physicians, using the medical dictionary for data filtering.

##### General

For all 12 languages, we use the translation and original data of Sharegpt and Alpaca. For Chinese, we additionally make use of WizardLM Dataset generated by GPT-4 based on WizardLM Method (Xu et al., ). For English, in addition to adding the WizardLM Dataset, we also add belebele to enhance multi-language reading comprehension capabilities, ai2_arc to enhance abstract reasoning capabilities, Capybara to enhance instruction following capabilities.

##### Web

For Chinese, we use the medical dictionary(^10^1010[http://thuocl.thunlp.org/#yixue](http://thuocl.thunlp.org/#yixue)) to filter out medical-related articles from the Wudao Dataset. For English, we use the English Medical
Vocabulary(^11^1111[https://www.nlm.nih.gov/research/umls/new_users/online_learning/LEX_001.html](https://www.nlm.nih.gov/research/umls/new_users/online_learning/LEX_001.html)) to filter out medical related articles in C4 Datase. For Spanish, we sampled 10% of CoWeSe Dataset and used the medical-eval-pt data for supplementation.
For Italian, we utilize data from it-medical-corpus and it-biomedical-dataset. For German, we use the German portion of data from opus-medical-de-en. For Russian, we employ data from medical-qa-ru-data, which contains 190,335 Russian Q&A posts from a medical-related forum. For Japanese, we use the medical web data from MedNLP-SC Social Media Corpus.

##### Math

For mathematical abilities, we choose MathInstruct, a composite dataset containing various mathematics-related tasks and problem formats.

##### Code

We choose Python-Alpaca and Leetcode-ZH-11k respectively to strengthen the ability to solve coding tasks in Chinese and English.

### A.3 Benchmark Across 12 Major Languages

To ensure the reliability of our evaluation, we use publicly available multilingual medical benchmarks consisting of multiple-choice medical questions, with accuracy as the evaluation metric. For languages with limited evaluations, following multilingual evaluation method of Llama3 (Dubey et al., ) we translate MMLU (Hendrycks et al., ) questions and answers using Google Translate. We employed a random selection of 3-shot queries to pose questions to the model, followed by answer extraction and evaluation of the model’s responses. Tab. 6 shows the sources of the 12 multilingual benchmark and the test data samples are shown in Fig. 6.

**Table 7: Benchmark across 12 Languages.**
| Language | Benchmark Composition |
| --- | --- |
| Chinese | MedQA-MCMLE (Zhang et al., ), Medical parts of CMMLU (Li et al., ) |
| English | MedQA-USMLE (Zhang et al., ), MedMCQA (Pal et al., ) |
|  | Medical parts of MMLU (Hendrycks et al., ) |
| Spanish | HEAD-QA (Vilares & Gómez-Rodríguez, ) |
| Russian | RuMedBench (Blinov et al., ) |
| Korean | KorMedMCQA (Kweon et al., ) |
| Japanese | IgakuQA (Kasai et al., ) |
| German | MMLU part of BioInstructQA (Labrak et al., ) |
| Portuguese | MMLU part of BioInstructQA (Labrak et al., ) |
| Italy | MedExpQA (Alonso et al., ), Translated medical parts of MMLU |
| Arabic | Translated medical parts of MMLU |
| Hindi | Translated medical parts of MMLU |
| French | FrenchMedMCQA (Labrak et al., ), Translated medical part of MMLU |

Figure: Figure 6: Sample English Evaluation Data (Similar to Other Languages).

Specifically, we follow Med-PaLM2 (Singhal et al., ) and select six subcategories in MMLU: Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, and College medicine. For MedQA, we choose the 4-options version. For CMMLU, we select seven subdirectories: Anatomy, Clinical knowledge, College medicine, Genetics, Nutrition, Traditional chinese medicine, and Virology.

### A.4 More Experimental Analysis on the Dataset

**Table 8: Performance of Diverse Models across 12 Languages, and the comparison before and after training of base models.**
| Model | Size | Ar | De | En | Es | Fr | Hi | It | Ja | Ko | Pt | Ru | Zh | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Closed-source Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gemini-1.5 Pro | - | 74.9 | 80.4 | 63.2 | 81.9 | 87.5 | 75.4 | 69.1 | 68.9 | 78.6 | 79.9 | 64.8 | 70.0 | 74.6 |
| GPT-4 Turbo | - | 79.8 | 84.7 | 68.5 | 85.8 | 89.7 | 72.8 | 80.9 | 77.5 | 79.5 | 87.0 | 78.1 | 68.9 | 79.4 |
| Claude-3 Opus | - | 78.8 | 87.2 | 69.8 | 87.9 | 90.7 | 78.6 | 84.6 | 82.1 | 85.2 | 86.7 | 80.1 | 70.9 | 81.9 |
| GPT-4o | - | 80.9 | 90.5 | 70.5 | 90.7 | 93.5 | 86.9 | 85.1 | 89.5 | 90.2 | 90.8 | 78.5 | 80.9 | 85.7 |
| GPT-4o mini | - | 74.1 | 80.7 | 70.5 | 82.3 | 84.7 | 75.2 | 77.7 | 76.4 | 77.2 | 82.2 | 80.9 | 67.9 | 78.2 |
| Open-source Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MMed-Llama-3 | 8B | 29.5 | 40.3 | 54.7 | 63.6 | 62.0 | 38.8 | 20.7 | 47.3 | 20.3 | 25.9 | 62.5 | 16.7 | 40.2 |
| OpenBioLLM | 8B | 27.5 | 58.1 | 49.6 | 59.2 | 53.3 | 44.4 | 41.5 | 39.1 | 46.7 | 27.0 | 64.8 | 41.8 | 46.1 |
| Llama3 | 8B | 33.5 | 55.9 | 48.4 | 60.5 | 58.4 | 48.0 | 39.9 | 38.9 | 49.5 | 51.5 | 63.3 | 43.6 | 49.3 |
| Base Models before and after Training |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Gemma2 | 2B | 34.6 | 39.5 | 47.7 | 46.7 | 37.1 | 37.7 | 32.4 | 30.3 | 34.2 | 38.3 | 62.5 | 36.1 | 39.8 |
| Aft. | 2B | 44.1 | 51.0 | 59.6 | 50.9 | 47.3 | 39.4 | 46.8 | 44.5 | 48.2 | 53.5 | 62.5 | 64.5 | 51.0 |
| Phi-3 | 3.8B | 18.3 | 41.0 | 43.4 | 40.0 | 40.8 | 19.3 | 34.6 | 17.0 | 18.1 | 37.8 | 57.4 | 22.6 | 32.5 |
| Aft. | 3.8B | 36.5 | 60.7 | 66.5 | 59.4 | 57.0 | 36.4 | 59.0 | 27.1 | 46.4 | 63.3 | 59.8 | 65.4 | 53.1 |
| Qwen2 | 7B | 45.7 | 54.0 | 58.6 | 60.5 | 58.6 | 35.4 | 44.7 | 45.4 | 52.6 | 48.0 | 77.3 | 81.5 | 55.2 |
| Aft. | 7B | 54.8 | 71.2 | 62.6 | 69.7 | 70.4 | 55.0 | 65.4 | 63.5 | 67.7 | 66.1 | 78.1 | 85.4 | 67.5 |
| Gemma2 | 9B | 60.7 | 69.9 | 59.1 | 71.0 | 71.3 | 63.2 | 70.9 | 58.0 | 66.3 | 72.6 | 71.5 | 56.5 | 65.9 |
| Aft. | 9B | 65.2 | 78.5 | 77.2 | 72.9 | 75.6 | 65.8 | 75.6 | 67.8 | 70.6 | 77.5 | 69.9 | 76.9 | 72.8 |

To further validate the quality and effectiveness of the data, we increase both model diversity and parameter count.

##### Experiment Settings

For Baselines, We select Gemma2-2B (Riviere et al., ), Phi-3-mini-4k (Abdin et al., ) and Qwen2-7B (Yang et al., ) as base models. We simultaneously selected the closed-source models, Claude3 Opus (Anthropic, ), and GPT-4o (Achiam et al., ), as well as highly competitive open-source models MMed-Llama-3-8B (Qiu et al., ), LLama3-OpenBioLLM-8B (Ankit Pal, ), and LLama3-8B (Dubey et al., ) as baselines. For Training, We use full-parameter fine-tuning with a learning rate of 1e-5 for the 7B and 9B models, and a learning rate of 1e-4 for smaller models.
All model use a batch size of 32 and a sequence length of 4096 for training on 8x NVIDIA A800-SXM4-80GB GPUs.

##### Result Analysis

As shown in Tab. 8, the performance of each model significantly improved after training with 12 major languages data, further validating the effectiveness of data quality. The fine-tuned models excelled in the medical domain, significantly surpassing open-source medical models with similar parameter counts. Additionally, the medical model fully fine-tuned based on Gemma2-9B performed nearly as well as large-scale closed-source models.

### A.5 Minor Languages Dataset&Benchmark

To assess the model’s generalization capability in multilingual contexts, we selected 38 minor languages with 12 major languages, totaling 50 languages.

- •
Major Languages English, Chinese, German, Portuguese, Spanish, French, Russian, Hindi, Italian, Korean, Japanese, Arabic
- •
Minor Languages Mongolian, Thai, Vietnamese, Lao, Malagasy, Cebuano, Sundanese, Ilokano, Dogue, Croatian, Galician, Czech, Corsican, Luxembourgish, Latin, Ukrainian, Bosnian, Bulgarian, Esperanto, Maithili, Serbian, Albanian, Slovak, Danish, Sanskrit, Norwegian, Guarani, Scottish Gaelic, Kurdish (Sorani), Maltese, Hebrew, Lingala, Bambara, Swahili, Sepeti, Igbo, Kinyarwanda, Hausa

##### Minor Languages Dataset

Given the extremely limited data for these minor languages, we extracted 2,000 samples from English data and used Google Translate to create training sets for each language.

##### Minor Languages Benchmark

For minor languages with limited evaluations, following multilingual evaluation method of Llama3 (Dubey et al., ) we translate medical-clinical part of MMLU (Hendrycks et al., ) questions and answers using Google Translate. We employed a random selection of 3-shot queries to pose questions to the model, followed by answer extraction and evaluation of the model’s responses.

## Appendix B General Experiment Setup

In this section, we describe the precise setup for our dense and MoE models. In this work, all our experiments follow the settings described below, with any specific settings mentioned directly in the main text.

##### Training

All model use AdamW, a batch size of 32, a sequence length of 4096 and a cosine decay learning rate schedule with a linear warmup of proportion 0.3 for training on 8x NVIDIA A800-SXM4-80GB GPUs. For model initialization and data sampling, we set the random seed to 42. For MoE models, We fine-tune them with the same settings as before after sparse upcycling from a dense model. For all routing strategy in MoE models, router parameters are initialized randomly with a zero-mean normal distribution with standard deviation 0.02. We set the learning rate of the dense model to 1e-4 and the learning rate of the MoE model to 1e-5.

##### Evaluation

To measure model performance, we extract the options from model output and calculate the accuracy with the reference answer. All evaluations use 3-shot examples. The optimal value is selected based on the average accuracy across three tests for each benchmark.

## Appendix C Prompts

### C.1 Prompts for Generating QA Pairs from Texts

Figure: Figure 7: Prompts for Generating QA Pairs from Texts. We show the English version of Prompt, and other languages are similar.

### C.2 Prompt for Generating Doctor-Patient Dialogues

Figure: Figure 8: Prompt Template for Generating Doctor-Patient Dialogues

## Appendix D Token Expert Routing Construction

Due to the varying number of tokens required for different languages, sentences of the same length may have different token counts depending on the language. We randomly selected varying amounts of data from benchmarks in different languages. After removing common English characters, the data was used as a test set for expert routing analysis, ensuring that each language had 80,000 tokens. Fig. 9 shows sample test data for several languages; the format is the same for other languages.

Figure: Figure 9: Test Data Example in Different Languages. Other Languages are Similar.

## Appendix E Different Languages for Hybrid Routing

The Hybrid routing distribution is shown in Fig. 10. It explains how the model allocates tasks of processing different languages among its internal experts, and how this allocation changes across different network layers.

Figure: Figure 10: Different Languages for Hybrid Routing.
Refer to caption: extracted/5921635/hybrid_new/ar_new.jpg

## Appendix F Complete Results on 12 Major and 38 Minor Languages

To comprehensively evaluate the performance of the proposed model across different languages, Tab. 9 presents the accuracy results on 12 major languages and 38 low-resource languages. We showcase the performance of MoE models with various routings (in Sec.3.1.2) and the Apollo-MoE series across different base model sizes (e.g. 0.5B, 1.5B, and 7B parameters in Sec.4) on these languages. The models achieve accuracy on par with that of the major languages, even for low-resource languages, highlighting the model’s strong generalization capabilities when handling resource-scarce languages.

**Table 9: Vital Model Perfrmances (Accuracy) on Major and Minor Languages.**
| Langs | Qwen2-0.5B | Lang-Spec. | Top2 | Hybrid2 | Apollo-MoE-0.5B | Apollo-MoE-1.5B | Apollo-MoE-7B |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Zh | 45.4 | 42.8 | 53.2 | 53.6 | 51.3 | 67.8 | 84.1 |
| Ko | 19.2 | 27.8 | 35.9 | 36.2 | 35.2 | 52.9 | 68.4 |
| Ja | 24.0 | 21.4 | 30.5 | 32.4 | 26.9 | 45.7 | 62.4 |
| Ne | 29.2 | 23.9 | 29.2 | 29.2 | 27.3 | 40.2 | 56.1 |
| Th | 33.4 | 32.2 | 26.1 | 28.8 | 39.0 | 45.5 | 59.8 |
| Vi | 34.1 | 27.1 | 30.9 | 29.1 | 35.9 | 45.6 | 61.1 |
| Lo | 34.1 | 27.3 | 28.0 | 30.3 | 34.1 | 37.5 | 45.8 |
| Mg | 30.7 | 22.7 | 27.7 | 34.1 | 32.2 | 41.7 | 46.2 |
| Ceb | 36.7 | 26.5 | 28.8 | 35.2 | 32.2 | 50.8 | 67.0 |
| Su | 32.2 | 30.7 | 33.0 | 35.2 | 34.8 | 48.5 | 67.8 |
| Ilo | 29.2 | 26.1 | 26.1 | 28.8 | 32.6 | 46.2 | 61.0 |
| Doi | 27.3 | 24.6 | 22.7 | 23.5 | 29.9 | 41.3 | 50.8 |
| En | 39.3 | 39.1 | 43.7 | 44.1 | 45.4 | 56.5 | 73.1 |
| De | 27.4 | 28.6 | 36.9 | 37.2 | 38.2 | 54.8 | 73.5 |
| Pt | 26.3 | 31.0 | 34.8 | 34.3 | 37.3 | 57.3 | 73.8 |
| Es | 32.9 | 33.3 | 38.9.0 | 40.0 | 39.8 | 53.5 | 69.4 |
| Fr | 21.3 | 16.5 | 39.9 | 40.7 | 38.4 | 53.3 | 72.4 |
| Ru | 46.9 | 52.3 | 58.2 | 58.8 | 64.1 | 72.9 | 74.2 |
| It | 20.7 | 24.5 | 37.7 | 38.9 | 39.9 | 54.4 | 71.9 |
| Hr | 32.6 | 26.9 | 31.8 | 34.1 | 35.6 | 48.9 | 67.0 |
| Gl | 36.0 | 23.9 | 37.9 | 38.6 | 40.2 | 56.8 | 71.6 |
| Cs | 34.5 | 26.9 | 32.6 | 32.6 | 36.4 | 54.2 | 67.0 |
| Co | 35.2 | 20.8 | 36.0 | 35.2 | 44.3 | 56.1 | 70.8 |
| La | 30.3 | 20.8 | 29.5 | 34.8 | 37.5 | 48.5 | 60.6 |
| Uk | 31.5 | 26.5 | 33.5 | 30.9 | 29.0 | 35.4 | 53.3 |
| Bs | 31.1 | 25.0 | 32.2 | 36.4 | 37.9 | 48.9 | 64.0 |
| Bg | 31.1 | 28.8 | 35.6 | 36.0 | 31.1 | 41.7 | 60.2 |
| Eo | 28.8 | 22.0 | 28.4 | 31.4 | 34.5 | 45.8 | 62.9 |
| Mai | 28.8 | 25.0 | 26.1 | 28.0 | 28.8 | 42.8 | 59.8 |
| Sq | 28.4 | 22.0 | 30.3 | 30.7 | 31.4 | 40.5 | 58.3 |
| Da | 33.3 | 25.4 | 30.7 | 36.4 | 36.7 | 55.3 | 67.8 |
| Sa | 29.2 | 28.0 | 27.3 | 30.3 | 25.0 | 40.2 | 54.5 |
| No | 33.7 | 25.8 | 27.3 | 31.8 | 33.7 | 47.3 | 67.8 |
| Gn | 30.7 | 25.4 | 28.0 | 36.0 | 34.1 | 45.8 | 56.1 |
| Sr | 30.7 | 28.0 | 33.0 | 34.8 | 17.8 | 33.7 | 39.4 |
| Sk | 34.1 | 27.7 | 35.2 | 34.1 | 38.3 | 50.8 | 67.0 |
| Gd | 31.8 | 23.1 | 29.9 | 30.3 | 33.7 | 45.8 | 50.8 |
| Lb | 35.2 | 24.6 | 28.8 | 30.7 | 38.3 | 54.2 | 67.8 |
| Hi | 32.9 | 33.3 | 38.9 | 40.2 | 39.8 | 53.5 | 69.4 |
| Ar | 27.3 | 28.8 | 34.5 | 35.1 | 36.3 | 47.2 | 58.3 |
| Ckb | 26.9 | 26.1 | 31.8 | 31.8 | 31.8 | 37.9 | 48.1 |
| Mt | 25.8 | 24.6 | 32.2 | 30.3 | 25.0 | 47.7 | 60.2 |
| He | 34.1 | 27.9 | 29.0 | 32.2 | 37.0 | 43.5 | 64.0 |
| Ln | 31.8 | 26.1 | 29.9 | 27.7 | 33.7 | 43.9 | 53.4 |
| Bm | 31.1 | 28.0 | 24.2 | 29.9 | 29.2 | 42.4 | 40.2 |
| Sw | 30.7 | 27.3 | 30.3 | 33.0 | 37.5 | 42.0 | 46.2 |
| Nso | 29.5 | 29.9 | 27.3 | 31.8 | 34.5 | 40.9 | 45.8 |
| Ig | 30.3 | 27.3 | 28.0 | 30.3 | 36.0 | 44.3 | 49.6 |
| Rw | 27.3 | 27.3 | 29.9 | 29.5 | 36.7 | 34.5 | 34.1 |
| Ha | 31.1 | 31.8 | 26.5 | 30.7 | 26.9 | 39.0 | 37.1 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: * According to Figure 2, Does the base model for the proposed method use a post-normalization Transformer instead of a pre-normalization one?


* This paper claims, "A major obstacle is the scarcity of medical data in many languages, limiting model development for underrepresented populations." According to Table 4, the average accuracies of major and minor languages among 50 languages in GPT-4o are 85.7 and 81.1, respectively. It appears that the performance of minor languages is not significantly worse compared to major languages in recent top models. This result seems to contradict the situation described in the paper. Do the authors provide any other explanation for the motivation behind developing the proposed method?


* The proposed method's best results, namely, an average accuracy of 69.9 for major languages and 58.3 for minor languages, as shown in Table 5, do not seem significantly better than the "dense" baseline, which achieved 69.0 and 55.7, respectively. Since I'm not familiar with the evaluation dataset used in this paper, I wonder if the differences of 0.9 and 0.6 for major and minor languages, respectively, are meaningful enough to consider.
soundness: 3
strengths: * This paper tackles the crucial issue of reducing language-related barriers to healthcare access for minor languages through LLMs.

* This paper proposes an intuitive and effective MoE approach for supporting multiple languages within a single MoE model.

* This paper provides an analysis of the inner flow of MoE gating.

* The experimental results are encouraging within the evaluation framework applied in this paper.
summary: This paper addresses the challenge of adapting medical large language models (LLMs) to low-resource languages, where data scarcity hinders equitable access to healthcare information. 
The authors construct a high-quality medical dataset covering 12 languages and develop a novel approach to model scalability using Mixture of Experts (MoE) modularity and a new routing method. 
Their analysis reveals a “Spread Out in the End” information flow mechanism, where cross-lingual information is focused in early model layers but diverges into language-specific pathways in later layers. 
This insight leads to the development of the Post-MoE architecture, which applies sparse routing in later layers to enhance model generalization and interoperability. 
Furthermore, by introducing "language family" experts, the model efficiently scales to 50 languages without adding extra parameters, making it both scalable and efficient for multilingual medical applications.
weaknesses: * The core idea of the proposed method does not appear particularly novel, as it resembles a combination of BTX and Sperse Upcycling, as discussed in the related work section.

* Although the results of the proposed method are consistently better than those of the baseline models with similar model sizes for dense models, MoE models have more parameters than the baseline dense models. Therefore, it may be unfair to directly compare the results between the baseline dense model and the MoE model with the same base model size. The authors should also provide results for dense models with model sizes comparable to those of the MoE models obtained by the proposed method.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Typo: "shwows"
soundness: 3
strengths: This paper tries to extend medical LLMs into different languages. The authors constructed a high-quality medical dataset and proposed a novel MoE routing method that employs language-specific experts and cross-lingual routing. The intuition of extending LLMs into different languages is very important, and the dataset is useful in the community.
summary: This paper tries to extend medical LLMs into different languages. The authors constructed a high-quality medical dataset and proposed a novel MoE routing method that employs language-specific experts and cross-lingual routing. The intuition of extending LLMs into different languages is very important, and the dataset is useful in the community. 
Several concerns about this research:
1. The paper focused on the medical LLMs, while the evaluation is based on MMLU; although it includes part of the medical-related tasks, the evaluation on this general domain dataset will bring additional noise on evaluating LLMs in medical tasks. More experiments on medical-specific benchmarks should be helpful to demonstrate the contribution of the medical dataset. Otherwise, we can only see the constructed dataset can contribute to the general domain but not specifically to the medical domain.
2. Much relevant research on extending pre-trained models to different languages is overlooked in this paper (related works). For example,  "GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost, IJCAI 2023" explores a simple way to extend LLMs to different languages. It would be better to compare the proposed method with other language extension methods.
weaknesses: Several concerns about this research:
1. The paper focused on the medical LLMs, while the evaluation is based on MMLU; although it includes part of the medical-related tasks, the evaluation on this general domain dataset will bring additional noise on evaluating LLMs in medical tasks. More experiments on medical-specific benchmarks should be helpful to demonstrate the contribution of the medical dataset. Otherwise, we can only see the constructed dataset can contribute to the general domain but not specifically to the medical domain.
2. Much relevant research on extending pre-trained models to different languages is overlooked in this paper (related works). For example,  "GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost, IJCAI 2023" explores a simple way to extend LLMs to different languages. It would be better to compare the proposed method with other language extension methods.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Fig 3b is really hard to read

 #params is more appropriate and apple-to-apple comparison with same active parameters comparing to dense models?
soundness: 3
strengths: Comprehensive experiments are conducted across models with varying parameter sizes (0.5B, 1.5B, and 7B), demonstrating scalability and superior performance among major languages.

The “Spread Out in the End” phenomenon is well-motivated and appears relevant for improving both interpretability and efficiency in multilingual models.
summary: The paper introduces a method to expand medical language models to cover 50 languages, focusing on those with limited data. Instead of scaling by adding experts for each language, the authors group languages into families, using shared experts to keep the model size manageable. They identify a key insight: early model layers handle cross-language information, while later layers focus on individual languages. 

Based on this, they propose a design that only applies sparse routing in the final layers, enhancing efficiency. Experiments show the model works well across both major and minor languages, suggesting it could improve access to medical information in diverse languages without requiring excessive data or computational costs.
weaknesses: Although the MoE model's efficiency is emphasized, a more extensive comparison with the same computation cost dense models (especially regarding computational cost and training efficiency) could strengthen the claims.

The improvement mostly appears in Italian and French, also decreased in many more low-resource languages. This seems to lead to some overclaim... how is the data-mixture determined? 

line 124: employing Google Translate to translate the questions and answers of MMLU - why not use mmmlu from OpenAI?

Have any experiments been conducted to assess the robustness of translations used for low-resource languages? If not, how do you propose to validate the accuracy of these translations, given the potential consequences in a medical context?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - [addressed] Considering that the language specialization of LLMs at the top and bottom layers was an arealdy-known phenomenon by early 2024 (e.g. "Do Llamas Work in English? On the Latent Language of Multilingual Transformers" or "How do Large Language Models Handle Multilingualism?"), do you still perceive that your "Spread Out in the End" analysis is bringing novel insights? In general, could you help me perceive the theorical aspects of your other contributions which I may have missed?
- [addressed] Did you release your multilingual test set to enable other works to evaluate against yours?
- [addressed] Do you intend to release your training data to facilitate replications of this work?
- What role does relative importance of languages play in the linguistic family grouping? English is likely overshadowing other languages using the same expert.
- [addressed] How did you choose your 50 languages? Moreover, why did you include Latin in your experiments? It's no longer an active language of science since many centuries, and Google Translate is unlikely to do a good job on modern medical documents in this case.

Not technical but still relevant:
- [addressed] Why do you want to present this work at ICLR? As a first-time ICLR review, I cannot make judgements about what type of works usually make an impact at ICLR, so I will leave this up to the AC Meta-Review. I will however raise my concern that this work doesn't seem to have a particularly strong tie to Representation Learning. It's also not particularly analytical or theoretical. While I personally would be very excited to discuss this further with the authors, I'm surprised to see this type of work submitted to ICLR instead of EMNLP or COLM. However, I see that ICLR also includes general machine learning as an accepted topic, so it's really difficult for me to judge this, and I won't. But I think the answer to this question might be helpful for the meta-reviewer.
soundness: 4
strengths: - Extensive and multi-layered analysis of cross-lingual pollination and the various trade-offs involved both during training and at inference time.
- High-quality design choices informed by serious and well-informed model analyses.
- Well-structured and information-rich paper.
- On top of that: an impressive new model for the medical domain.
summary: In this paper, the authors detail with great precision the training procedure of their Mixture-of-Language-Family-Experts LLM for the medical domain, which combines several existing techniques in a novel way. The authors balance the benefits of cross-lingual learning (making sure low-resource languages benefit from knowledge only written in high-resource languages) with the curse of multilinguality (cross-lingual contamination of linguistic structures) using linguistic experts which always activate for their respective language but can also activate for other languages at the router's discretion, thereby ensuring that sentences in the train set benefit multiple languages simultaneously without impacting all aspects of every other languages. An evaluation is carried for 12+ languages (up to 50 in the scale-up experiments).
weaknesses: - [addressed] Evaluation is performed on translated test sets, generated by other language models.
- [addressed] Little discussion is made of the risk of test set contamination in the training set of the model in languages other than the source language. The 64 token overlap used breaks down once the input data is multilingual, since translations have no or minimal token overlap.
- Most techniques presented in the paper do not appear to be very novel, even though their combination seems to be, and their choices are well-justified. Still, it's sometimes tough for me to determine whether this is really an academic work or just a high-quality engineering report written in paper form.

Meta-review: This paper tackles the task of adapting medical LLMs to new languages in order to increase access to these technologies. To do this, the work first introduces a new medical QA dataset for model tuning in 12 languages; this dataset is automatically converted from text to QA format with ChatGPT and then used to finetune a dense LM. They then scale up their setting using a novel MoE architecture (Post-MoE) and routing method, based on findings from their analysis of multilingual information flow within the models. The multilingual medical MoE model is shown to work well in both "major" (I assume this means high-resource?) and "minor", or low-resource, languages, suggesting that this is a promising direction for multilingual adaptation of LLMs.

Strengths:
- The paper addresses an important issue (kMZ, somb) and provides a well-designed approach for tackling it with MoE models (kMZ, CPZy), as well as a detailed analysis of the method to better understand how it works (kMZh, CPZy).
- The authors construct a new medical dataset for model tuning, which fills a currently open niche in terms of instruction tuning datasets (somb).
- The paper is well-written and easy to follow (CPZy). 
- Comprehensive experiments (sQUP), particularly after adding many experiments to address reviewers' concerns, including more evaluations using a new dataset, MMLU, and additional baselines with comparable parameter counts to the MoE models.

Weaknesses:
 - The experimental results show relatively small gains in performance, even for minor/low-resource languages (kMZh). The gains across languages are uneven (some languages benefit much more than others) without an analysis of why this occurs (sQUP).
 - Similarly, the grouping of languages by "language family" is quite coarse-grained and doesn't take into account the many factors (script, grammatical similarity, vocabulary overlap, etc.) that are known to affect cross-lingual learning in LMs (e.g., [1]). The scaled model would likely work much better, particularly for the low-resource languages, if languages were more carefully selected and grouped into experts.
 - A minor note, but using "major" and "minor" as language descriptors is uncommon in multilingual research. I recommend changing this to be "high-" and "low-resource" languages, or adding an explanation for why this terminology is used instead.

[1] Chang et al., When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages. https://arxiv.org/abs/2311.09205

Decision: Accept (Poster)