Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts
Authors: Junmo Kang
Leonid Karlinsky
Hongyin Luo
Zhen Wang
Jacob A Hansen
James R. Glass
David Daniel Cox
Rameswar Panda
Rogerio Feris
Alan Ritter
Venue: ICLR 2025 Poster (2025)
Keywords: Efficient Specialization of LLMs
Self-Improving
Mixture of Experts
Primary area: foundation or frontier models, including LLMs
TL;DR: We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts using minimal resources.

Abstract: We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts, named MiXSE (MiXture of Self-specialized Experts). Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data, each equipping a shared base LLM with distinct domain-specific capabilities, activated via self-optimized routing. This allows for dynamic and capability-specific handling of various target tasks, enhancing overall capabilities, without extensive human-labeled data and added parameters. Our empirical results reveal that specializing LLMs may exhibit potential trade-offs in performances on non-specialized tasks. On the other hand, our Self-MoE demonstrates substantial improvements (6.5%p on average) over the base LLM across diverse benchmarks such as knowledge, reasoning, math, and coding. It also consistently outperforms other methods, including instance merging and weight merging, while offering better flexibility and interpretability by design with semantic experts and routing. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.

Published as a conference paper at ICLR 2025
SELF-MOE: TOWARDS COMPOSITIONAL LARGE LAN- GUAGE MODELS WITH SELF-SPECIALIZED EXPERTS
Junmo Kang"
Leonid Karlinsky
Hongyin Luo MIT
Zhen Wang UCSD
Georgia Tech
MIT-IBM Watson AI Lab
Jacob Hansen MIT
James Glass
David Cox MIT-IBM Watson Al Lab
Rameswar Panda MIT-IBM Watson Al Lab
MIT
Rogerio Feris
Alan Ritter Georgia Tech
MIT-IBM Watson AT Lab
ABSTRACT
We present Self-MoE, approach that transforms monolithic LLM into com- positional, modular system of self-specialized experts, named MiXSE (MiXture of Self-specialized Experts). Our approach leverages self-specialization,, which constructs expert modules using self- geffraenertted synthetic data, each equipping a shared base LLM with distinct domain-specific capabilities, activated via self- optimized routing. This allows for dynamic and capability-specifie handling various target tasks, enhancing overall capabilities, without extensive human- labeled data and added parameters. Our empirical results reveal that specializing LLMs may exhibit potential trade-off's in performances on non-specialized tasks. On the other hand, our Self-MoE demonstrates substantial improvements (6.5%p on average) over the base LLM across diverse benchmarks such knowledge. reasoning, math, and coding. It also consistently outperforms other methods, in- cluding instance merging and weight merging, while offering better flexibility and interpretability by design with semantic experts and routing. Our findings high- light the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.
INTRODUCTION
The remarkable success of Large Language Models (LLMs) has been largely attributed to their gen- eralist nature, allowing them perform wide variety tasks (Brown al. 2020 Touvron et al.) 2023 Jiang et al., 2023 Team et al., 2024). Predominantly designed monolithic architectures, these models rely extensively on large-scale data to embed generalized language capabilities across vast parameter spaces. While effective, this monolithie architecture, as illustrated in Figure T in- herently suffers from significant trawbacks such as inefficiency in scaling (Zhang et al. 2024 Wan let al.) 2024), susceptibility to forgetting previously learned information when adapted to special- ized tasks (Kotha et al.) 2024; Huang et al. 2024 and a lack of transparency which leads to the black-box nature (Zhao al. (023)..
Meanwhile, the increasing demand to handle domain-specific or expert-level tasks has highlighted the need for specialization LLMs (Cheng et al., 2024 Ling et al. [2033] Feng et al. 2024). How- ever. effective tuning often relies on high-quality.
-nnonttdd data, which is costly and chal- lenging to scale (Kang al. 2023), especially in specialized domains where expertise is searce and valuable (Wu et al.) 2023). Self-specialization (Kang et al.) 2024) offers a promising alternative, aligning models with self-generated synthetic data. While this technique has proven effective in cross-task generalization within target expert domain, we posit that it may compromise perfor- mance in areas outside the target domain.
"Correspondence junmo. kang@gatech..ed

---

Published as a conference paper at ICLR 2025
Monolithic
Compositional
Base
Specialized LLM (Knowledge)
MOXSE
Gereralitt
Spestalar
Stecistrrts
54.
61.3
$2.5
943
AT
417
40.1
Salf-Spacializatin
Self-MoE
Knowledge
Ressoring
Mach
Coding
Figure/ I: Concept of Self-MoE, illustrating the transformation from a monolithic LLM to a compo- sitional system, MiXSE, without extensive resources and addition of significant parameters. MiXSE distinguishes itself from traditional MoEs and other models in post-training, lightweight semantic experts, and/or self-generated synthetic data. The results showcase MiXSE's improved capabilities over the base LLM (e.g.. Gemma-7B across all domains, unlike the knowledge-specialized LLM that compromises other capabilities.
In this paper, we explore the following question: How can we build compositional LLMs that enjoy versatile expertise, while using minimal resources? We introduce Self-MoB (Figure T), approach that transforms a monolithic model into a compositional (Zaharia et al., (024) system, called MiXSE (MiXture of Self-specialized Experts). This approach differs from prior MoË work using LoRA (Hu et al., 2022), which either relies on human-labeled data (Wu et al. 2024) or assumes the existence of trained modules (Huang et al. 2023) Mugeeth al.) 2024). Instead. our ScIf-MoB constructs individual lightweight expert modules from scratch using synthetic data, inspired by the concept of self-specialization.. Each module is integrated with a shared base LLM, and the entire system is enhanced by self-optimized routing mechanism. In contrast to monolithic models, which often suffer from forgetting issues when adapted or merged under fixed, static parameters, our modu- lar design preserves the integrity and semantics of each expert. This allows for dynamic, precise handling of various target domain tasks, boosting the model's overall capability, adaptability, and interpretability.
Through extensive empirical studies conducted across a variety of popular domains, including knowledge, reasoning, math, and coding, find that specialization often comes with trade-off's, typically degrading performance non-targeted domains. However, our Self-MoE demonstrates substantial overall improvements performance on other tasks. Notably, the compositional nature of our MiXSE appears to exploit synergies among experts, even outperforming all individual specialized experts.
base LLM across all target domains without compromising
Moreover, MiXSE clearly surpasses other strong baselines such as instance merging and weight merging, under similar settings, while offering better flexibility and interpretability. Detailed anal- yses highlight the critical role of the routing mechanism and the contribution of semantic experts in achieving these results. Our interpretable visualizations of routing distributions further elucidate how tasks are dynamically allocated to the most relevant experts. Lastly, we further validate that there are no issues related to forgetting unlike monolithic baselines, and that our approach can be applied to various model families and sizes. In summary, our key contributions are as follows:
We highlight the inherent limitations of monolithic model specialization, where focusing on a specific capability often comes at the cost of degrading performance in other domains.
We propose Self-MoE, which allows a base, monolithic LLM to upgrade into a modular system of lightweight, self sifesspecillized experts, without requiring extensive human supervision, compute resources, or overhead in active parameters.
We provide comprehensive experiments and analyses across a range of benchmarks, where Self- MoE demonstrates consistent improvements with average of 6.5%p across domains over base LLM, outperforming various baselines. Our ablation studies validate the impact of modularity. routing strategies, and the of self-generated synthetic data. Moreover, analyses explore routing distributions, forgetting issues, and the applicability to five different base LLMs.
2 PROBLEM STATEMENT
The primary focus of this work is on self-improving LLMs' target capabilities on the fly, specifically under settings constrained by minimal resources and without the addition of significant parameters.
2

---

Published as a conference paper at ICLR 2025
Self- lelflppecialization
MiXSE (MiXture Self-Specialized Experts
Targeted Generation
Nk
Bone LLM
Instruction: iine soud 87 of coeep
Specialiong and Pepert's
sarred from
on?
G Sel. Special Aa
Sel--lign with LoRA, /
Suoeedion Operr's
:u T) (20,,
dl c A8,
Rocker d.
D- U
Figure 2: Overview of the Self-MoE approach to building compound system of specialized experts and a router in a self-improging manner. In the Self Specializaaii phase (left side), the base LLM is aligned with self self-generated synthetic data for each target specialization, producing lightweight expert modules. The right side shows MiXSE where each self-specialized expert is dynamically engaged based on the decisions of the self-optimized router.
Traditional LLMs, which are generally monolithic, require expensive human-labeled data to be bet- ter specialized, thereby limiting their adaptability and scalability when resources are constrained. We hypothesize that modular, compositional model utilizing self-generated synthetic data for self- improvement can dramatically improve specific target capability, adaptability, and interpretability while reducing dependency on expensive Homan-annotatted datasets.
Specifically, given a base LLM O₂ and a minimal set of seed data (e.g., 100) for each of the target capabilities (T (e.g., knowledge, math), our goal is to transform O₂ into an enhanced composi- tional model O comp where n target expert modules { the Self-Mot transformation function is defined as:
)a are effectively integrated. Formally,
ftrnss (9u: (T.H.)) camp Ou U )
Here, under our problem setting, the number of parameters of O₂ and Ocomp should not be signif- icantly different, necessitating that the expert modules AO; be lightweight (i.e.. LoRA (Hu et al.) ). The available seed data are limited but can be reasonably collected (e.g., 100). Importantly, we not assume the availability of larger/teacher models at one's hand; instead, we aim develop method that enables self-improvement and is designed to be universally applicable.
METHOD: SELF-MOE
In this section, describe Self-MoE, our proposed framework designed to build compositional model in which specialized expert modules and a routing component are learned in a self-training manner cooperate effectively. At high level, Self-MoE decomposes the monolithic structure of a base LLM into dynamic mixture of self-specialized units, each equipped with distinct target capabilities. This section outlines the overall pipeline and architecture of Self-MoE, illustrated in Figure 2 which details both the self-speecalizzation of individual target expert modules and their integration to form a compositional system, MiXSE (MiXture of Self-specialized Experts).
3.1 BUILDING EXPERT MODULES THROUGH SELF-SPEECIALIZATIOON
The first step of Self-MoE creating specialized modules | for each target expertise, while adhering to the desiderata discussed in Section 2] That is. the modules should be lightweight and
self-improving. We employ self-specializattion (Kang al. 2024) where base LLM is aligned with
self-generated data for target specialization, resulting in lightweight LoRA (Hu et al. 2022 experts.
Targeted Generation. Self-specialization involves generating synthetic instruction-respons data D, ((nss (1) (2)), (inst(") resp!")) (2) )..} tailored each target domain T,, We ensure the data
3

---

Published as a conference paper at ICLR 2025
is both diverse and highly relevant to the specialized tasks/domains each module will address. The generation includes the following steps:
(1) Seed Construction: First, given target T; identified. prepare small number of seed ex- amples (c.g., 100) that capture essential characteristics and scenarios relevant to each target domain T- While exploit existing datasets for the purpose of demonstration, we posit manual annotation for such a small number should be reasonable in real-world applications. These seeds serve as the foundational dataset from which synthetic variations are generated.
(2) Instruction Brainstorming: Once the seed examples are established, the next step is to diver- sify the range of instructions (and corresponding input contexts) through brainstorming process. Specifically, we promp]] base LLM 4 to create new instructions following sequences of seed instructions given in-context.
The final step involves generating corresponding responses for the newly created instructions. We use seed instruction latent relevant knowledge from Op.
(3) Response Generation:
seerructoo-resooseee response pairs as in-context demonstrations extract
Self-Align with LoRA With each specialized synthetic data Dj in place, we now proceed with the self-alignment of O, to induce specialization, separately producing lightweight expert components AO, Note that D; are self- seff-generated by O, and used to specialize the same On using an adapter module A0,, resulting in an specialized model θ Ospec Ou + A Specifically, utilize Low- Rank Adaptation (LoRA) (Hu et al. (022), which integrates additional trainable parameters that are specific each domain T; while keeping O, intact. Within the corresponding θ, define 0 as the weights at a certain layer where LoRA is attached. Let 0 Ospec E Baak be updated weights at a specific LoRA layer which can be decomposed as:
0 Opec 0o + +0;
= 00 + 00,
where G3 E Edxrank and A E Prankx with rank < min(d, k). The forward pass becomes;
h = Dspec = 60x + AB, 0A.
This applies to all LoRA layers, and only ΔQ; = (A _!!"" ...) is updated during training using D.. As whole, this process of seff-specialization can be defined producing expert module A0, for the i-th target along with the corresponding synthetic data D; (Left in Figure
fex (0o, Ti) - (AQ;, D.)
We iterate this process for each target domain, focusing knowledge, reasoning, math, and coding.
3.2 MIXTURE OF SLLL-SPECIALIZED EXPERTS
After each expert module is individually specialized through the self-specialization process, they are integrated into a compound system O comp. MiXSE (MiXture of Self-specialized Experts). MiXSE is designed to leverage the distinct capabilities of each module, orchestrating their cooperation to handle diverse tasks dynamically and efficiently. To achieve this benefit, router module 6, is also incorporated, which analyzes each input token to dynamically route to the most appropriate expert module based on the task at hand.
Specifically, within each layer, the output h for each input is calculated by combining the contri- butions of the selected expert modules A0,, weighted by their relevance determined by the router:
h 00x
a; A;A0
0,x
a, AA.
where a represents set of weights computed by the router (i.e., linear layer) 0, Raxk
= too-k(sof
'The prompts can be found in Table 11 14
Appendix.
4

---

Published as a conference paper at ICLR 2025
Note that we only take top-k probabilities and mask out the others efficiently reduce computation. In essence, this also allows the pre-trained base weights By to be sufficiently able to contribute, mitigating potential issues of over-specialiizaiion such as forgetting
diminished generalizability. The router 0, is a linear layer, shared across all LoRA layers, and is trained using the aggregated self-generated data D (Di) to learn how to optimally select modules for given task:
[(,,) = - (issr repg)_ (resp) inst; 8,, ( (A)
It can be noted that the router is not provided with explicit supervision about which expert should be selected for each token, as there is no fixed label for each token indicating the correct single expert. Instead, it learns the optimal expert selection indirectly through training instructions and responses. The supervision comes from the responses, where the routing decisions are determined dynamically based on the token-level features allow the model produce better responses. Importantly, the router is optimized separately after the expert modules are trained and frozen, ensuring the explicit semantic distinction of the expert modules is preserved.
self-generated
EXPERIMENTS AND RESULTS
Datasets. We evaluate Self-MoE across diverse domains categorized into knowledge, reasoning, math, and coding: MMLU (0- & 5-shot) (Hendrycks et al.) (021a), BBH (3-shot) (Suzgun et al.) 2022 GSM8K (8-shot) (Cobbe et al. 2021), and HumanEval (0-shot) (Chen et al.) 202T), respec- tively. For MMLU, we primarily employ the O-shot setting unless otherwise specified, based on established observations (Dettmers et al.] 2023) et al.) (024)) that tuning yields only marginal ef- fects in the 5-shot setting for this task. To test generalization (Section 4.4). we additionally evaluate on MATH (4-shot) (Hendrycks et al.) 2021b), MBPP (3-shot) (Austin et al.. 202T), NaturalQues- tions (5-shot) KWwitkowski et al. 2019). TriviaQA (5-shot) (Joshi et al. 2017). Hellaswag (0-shot) [Zeller et al. 2019), PIQA (0-shot) (Bisk et al. 2020), and TruthfulQA (0-shot) (Lin et al. 2022
Baselines. To assess the effectiveness of Self-MoE, we compare performance against several base- lines that are similarly trained using LoRA and that use the same number of active parameters during inference for fair comparisons:
Four Self-Specialized Models (Kang et al. 2024): Trained on self-generated synthetic data for individual domains: knowledge, reasoning, math, and coding.
Instance Merging (Multi-Task Tuning) (Chung et al. 2024): Leverages the aggregated synthetic data generated by self-specialization to train a model capable of handling multiple tasks.
TIES Yadav al. 2023). DARE YYu al. 2024): ; Advanced weight merging methods integrating
multiple expert strengths into unified model.
Note that Self-MoF does not require the base models to be implemented using specific architec- tures. Rather, Self-MoE builds upon purely any base LLMs using LoRA-based fine-tuning like other baselines, which ensures fair and consistent comparisons. We also contextualize these results with
computationally intensive methods reported in the literature, despite indirect comparisons: BTM LLi
jet al., 2022). Sparse Upcycling (Komatsuzaki et al, 2023), BTX (Sukhbaatar et al., 2024). GLAN
(Li et al. 2024a), Orca (Mitra et al. 2023), and Merlinite (Sudalairaj et al. 2024
Appendix D.I
Implementation Details. We adopt Gemma-7B (Team et al. 2024) as a base LLM for our main experiments, and additionally apply Self-MoE various models, such as LLaMA-2 7B & 13B (Tourron et al.; 2023). Mistral 7B (Jiang et al. 2023). and LLaMA-3 8B AAI@Metal 2024) in Section 4.5, We use 100 seeds to generate 5K synthetic data for each domain, resulting in 20K data. Each LoRA module contributes less than 0.3% to the parameters of the base model, and the router's parameters are negligible, resulting the added parameters of MiXSE amounting to only about 1 1%.
4.1
MAIN RESULTS
In Table I we showcase comparative benchmark results of various approaches across four special- ized domains: knowledge, reasoning, math, and coding. All baselines use self-generated synthetic data based the same Base LLM, Gemma-7B, and LoRA for tuning ensure fair comparisons.
5

---

Published as a conference paper at ICLR 2025
Table
Main results. All models built upon the hase L.M,, Gemma-7B, taking self-improving approaches and having the same active parameters during inference. Corresponding aligned performances of self-specialization are underscored. Each column's 's best performance highlighted in bold, while the gains achieved by our MiXSE over the base LLM are indicated.
Active Params
Knowledge (MMLU)
Reasoning (BBH)
Math (GSMSK)
Coding (HumanFval)
Method
Avg.
Base L.M
7B
58.4
56.1
42.5
34.1
47.8
Specialized LLM Each Capabiility
Knowledge Self-Spec.
64.0
41.7
40.5
28.0
43.6
Reasoning Self-Spec.
60.1
60.2
41.0
28.7
47.5
Math Self-Spec.
0.3%
59.3
58.9
50.0
36.0
51.1
Coding Self- Self-Spec.
0.3%
57.2
57.2
46.0
37.2
49.4
Merging Methods
Instance Merging
7B + 0.3%
62.6
57.6
53.5
36.0
52.4
TIES Merging
7B + 0.3%
63.7
56.3
38.5
32.9
47.9
DARE Merging
7B + 0.3%
37.7
59.6
45.0
34.8
44.3
MiXSE (Ours)
7B + 0.3%
65.6
1.7.2
61.1
1.5.0
52.5 1 10.0
37.8
3.7
54.3
1.6.5
First, confirm seff-sppcialiization markedly enhances target-specific expertise, compared to the base LLM. For instance, we can see substantial gains from corresponding specialized models (e.g., Knowledge Self-Spec. in the knowledge domain): 58.4 to 64.0 in knowledge, 56.1 1 to 60.2 in rea- soning, and so on. However, this focused improvement sometimes comes at the cost of reduced performance in non-targeted areas, evidenced by the drop scores for the Knowledge Self-Spec. model in reasoning, math, and coding. This trade-off highlights the inherent limitation of over- specialization. contrast, our MiXSE, demonstrates consistent improvements across all domains, due to its modular, compositional architecture that makes use of dynamic routing to leverage opti- mal experts. Surprisingly, even outperforms all corresponding specialized models, indicating that it effectively synergizes the strengths of each specialization.
comparison with other static merging methods like Instance Merging, TIES, and DARE, MiXSE stands out for its superior adaptability. While they attempt to combine the strengths of different spe- cialization areas into a single model, they lack the dynamic flexibility that MiXSE offers. Notably, simple instance merging (i.e., multi-task tuning), though effective in enhancing the base LLM across domains, still falls short of achieving the superior average performance 54.3 seen with MiXSE. This validates the advantages of dynamic expert integration in compositional system.
4.2 ABLATION STUDY
Now that we have verified the effectiveness of MiXSE as a whole, we evaluate the impact of different configurations and components of the system, presented in Table [ The configurations vary in terms of routing strategies and integration of experts, offering insights into the contributions of each element the system's overall effectiveness.
We start by examining the Top-k routing strategy, which plays crucial role in our model. Our findings show that both the Top-1 and Top-2 expert configurations deliver the best performance. This suggests that identifying and leveraging the most relevant expert for a given task is typically sufficient and most effective. On side note, the similar performances the different configurations may highlight the robustness of our method. Given the similar performances, we prefer the Top-1 expert setup for better efficiency.
Interestingly, the results also indicate drop in performance when using All Experts. This can be attributed to that involving all experts regardless of their relevance can introduce noise and dilute the specific contributions of the most pertinent experts. Additionally, involving more experts than necessary can increase computational overhead.
We observe that the performance significantly decreases with random routing (i.e., w/o Self- Optimized Router), highlighting the router's role in dynamically tailoring the selection of experts according the specific requirements of each task. The router's ability to discem and activate the

---

Published as a conference paper at ICLR 2025
Table 2: Analysis and ablation of the router in our MiXSE. Configurations vary to investigate the optimal number of experts used, to verify the possibility of self-learning for the router, and to see the importance of semantic distinctions among experts within the compositional system.
Configuration
Knowledge (MMILU))
Reasoning (MBS))
Math (GSMBK)
Coding
Avg.
(HumonEval)
Base LLM
58.4
56.1
42.5
34.1
47.8
Top-k Routing
w/ Top-1 Expert
65.6
61.1
52.5
37.8
54.3
w/ Top-2 Experts
65.5
60.9
52.5
38.4
54.3
w/ All Experts
65.4
58.9
54.0
33.5
53.0
Romiing Strategy
w/o Self- Seff-Optimized Router
59.9
58.5
48.0
36.6
50.8
w/o Shared Router
59.5
59.1
50.5
32,9
50.5
Experts Router Joint Training
w/o Semantic Experts (Top-1)
64.5
58.1
46.0
33.5
50.5
w/o Semantic Experts (Top-2)
64.2
53.3
48.5
36.5
50.6
most suitable experts based on the context critical for optimizing performance. Notably, this ability is learned by relying on very small amount of seed data. When employing layer-specific routers instead the shared router, we found that the performance substantially drops, despite hav- ing about 200x more parameters, justifying our choice. This might be attributed to the fact that the layer-specific ones may introduce conflicting routing decisions, possibly requiring more data or hyperparameter tuning to become effective.
Another interesting finding comes from the configuration where experts and the router are jointly trained, which means that the semantic distinctions among experts may be diluted. This setup (w/ either Top-1 or Top-2) substantially decreases performance relative to scenarios where the router and experts are optimized independently. This decline validates that semantic experts play crucial role in enhancing the system's 's capability to handle tasks requiring specific expertise, while offering better interpretability (Section
4.3 ROUTING ANALYSIS
Understanding how MiXSE allocates tasks to its various experts is crucial for gauging its in- terpretability. By analyzing the routing distri-
1.8
Knowledge (MMLLU)
Reasoning (BIHH)
touting Weggta
0.-
0.
0.5
06
butions across four distinct domains, we aim to see whether the system matches queries to the most suitable experts. Figure presentts the routing distributions used to solve each bench- mark, where the weights are averaged across to- kens and layers within individual tasks.
14
Expart
Ixpar
Oppen
Expert
Expert
Kapert
Exprr
Deper
1.0
Math
Coding (HumanEval)
Routing Weighte
(GSMBK)
We first observe that the MiXSE's router ef- fectively selects the correct expert for each corresponding target. This is evident from the impressive alignment between tasks and the experts chosen by the router, for exam- ple, the knowledge expert predominantly han- dles knowledge tasks, while the coding expert is routed coding tasks. This highlights the router's ability to learn and apply this routing automati- cally and consistently, making the system's 's decisions interpretable and trustworthy.
Cucobedge
Math
Caoling
Krowiedgee
Reassning
Marth
Codree
Figure 3: Routing analysis that shows routing dis- tributions over four domains for each benchmark, averaging the weights across tokens within indi- vidual tasks.
Beyond the direct matching of tasks to domain-speciff experts, the router also demonstrates its abil- ity to exploit synergies between different areas of expertise. For instance, the reasoning expert is frequently involved in tasks across the knowledge. math. and coding, reflecting the system's 's com- positional use of expertise. This explains the reason for MiXSE's superior performances across all domains even beyond all specialized modules in Table

---

Published as a conference paper at ICLR 2025
Table 3: Investigation on generalization and a forgetting issue Self-MoE. Von-Targe (In- Expertise) indicates where MiXSE does not directly specialize using seed data directly while relevant to targets. Non-Target (Out-of-Expertise refers to irrelevant cases.
Category
Benchmark
Instance Merging
Target
Actderiic Knowlodge
MMLU
62.6
Reasoning Math
BRH
56.1
57.6
61.1
GSMBK
42.5
53.5
52.5
Target Average
47.8
52.4
Non- Turger (le-Fiteerteee
MATH
15.3
Coding
MBPP
37.8
37.6
39.6
Now Target (fut-af Expertise)
Knowledge
Natural Questions
TrivisQQA
58.6
62.5
Commonsense
Hellaswan
PIQA
81.1
N0.6
80.1
78.0
80.7
81.2
Salety
TruihfolQA
42.2
Non: Non-Target Avenage
50.4
47.7
50.6
4.4 GENERALIZABILLTTY TEST
While Self-MoE has shown clear benefits in target benchmarks such as MMLU, BBH, GSM8K, and HumanEval, one may be curious about its generalizability to non-targets, or concerned with the potential issues of specialization such as forgetting. In Table 33] we conduct an investigation using non-targeted benchmarks that were not utilized in building MIXSE.
On MATH and MBPP benchmarks, which can be considered highly relevant to target benchmarks, GSM8K and HumanEval, we find our Self-MoE can still improve over the base LLM even though they were not directly targeted in our training regime, indicating generalizability.
Concerning the potential side effect of forgetting, we extend our testing to include domains such as world knowledge, common sense, and safety, which are rarely associated with the targets directly. Our experiments show that overall, there are rarely meaningful performance drops when applying our Self-MoE. Only a minor drop is observed with MiXSE in TriviaQA, but this is substantially less than in the case of instance merging. This suggests approach almost maintains existing knowl- edge for non-targets while significantly boosting target performances, unlike monolithic baselines.
4.5 APPLICABILITY TO OTHER BASE LLMs
Following the successful demonstration of our Self-MoE approach based on Gemma-78 7B, we now present Figure where we apply Self-MoE
Merging
Avg. Performeroo
to other base L.MSS beyond Gemma-7B. We use diverse model variants including LLaMA-2
350
361 30.
7B & 13B, Mistral 7B, and LLaMA-3 8B. Our findings suggest that our approach improves all models on average regardless of the model fam-
Gemma 78
LLaMA-278 78
LLaMA-2 138
Misnalle 78
LLaMA-3 88
Figure Results of Self-MoE w/ other LLMs.
ily, size, and level of base performance, outperforming the strong instance merging baseline.
4.6 IMPACT OF THE NUMBER OF SYNTHETIC DATA
Figure 5 illustrates the impact of scaling self-generated synthetic data for Self-MoE. As the data scales from 0 to 20K, our MiXSE model demonstrates substantial and consistent improvements over the base one average per- formance across domains, suggesting the scalable poten- tial of Self-MoE. Instance Merging, serving as a strong baseline, also benefits from increased data, but the gains progress at a slower rate, as evidenced by linear trend- lines. This reflects the inefficiency of the static merging scheme, which, being monolithic, suffers from trade-offs in knowledge gains and forgetting.
Mosse
Performanca
... Trendline
Instance Nerging
351
Trendine
(ixitaco
Baxn
Syntheri
Figure 5: Analysis with the varied sizes of self-generated synthetic data.
8

---

Published as a conference paper at ICLR 2025
4.7 SCALING THE NUMBER OF EXPERTS
In Table 4 present the results of MiXSE composed of varying numbers experts, with experts added progressively one at time in the order of knowledge, reasoning, math, and coding. The re- sults indicate that starting with the knowl- edge expert, which initially exhibits a per- formance trade-off, subsequent additions
Table 4: Scaling the number of experts. K: Knowledge expert. R: Reasoning expert. M: Math expert. C: Coding expert.
Experts
Knowledge (MMLU)
Reasoning (BBII)
(GSMBK)
Coiling (HumanEval)
Arg.
LLM)
58.4
((K)
64.0
(K+R)
65.8
58.0
43.0
32.3
19.8
of reasoning, math, and coding experts consistently enhance overall performance.
(K+R+M)
(K+R+M-CC
62.7
65.6
61.1
61.5
52.5
54.5
37.8
32.9
54.3
52.9
This highlights the compositional MiXSE's advantage of adaptability and modularity.
4.8 ANALYSES ON SELF-GENERATED SYNTHETIC DATA
We conduct analyses of the self-synthesized datasets in Ta- ble [5] For diversity measure- ment, we first analyze the lin- guistic diversity using Type-to- Token Ratio (TTR), and the se- mantic similarity of the pairwise instructions' embeddings using SBERT (Reimers & Gurevych] 2019). Synthetic data demon- strates comparable linguistic di- versity to human-labeled data, with slightly higher TTR for BBH, suggesting that the syn- thetic data includes richer lexical variation, especially in reasoning tasks. For semantic similarity, synthetic data achieves generally low similarity within each dataset, similar to human-labeled data (0.3307 vs. 0.3279) on average. This suggests high semantic diversity overall, reflecting the natural diversity found in human- data.
Table 5: Analyses of self-generate synthetic data in terms of diversity, complexity, and naturalness.
Knowledge (MMLU)
Reasuning (BBI1)
(GSMBK)
(Tumanlaval)
Avg.
Tar- Tales Ratse
Labeled Synthetic
0,2639
0.1743
Saaleriy
Humon- Labeled Deta
0,2625
0,175
04125
0.4688
0,3279
Synthric Dats
0.3179
1948
0.350
0:4791
0,3307
Chongeenis Accanscy
L.M a-sejjdge (GPFT-4o)
$5.0
68.0
60.0
50.0
58.3
Modd Peyfonsace wsing Difirest Pare
Syntheii
Synthtic
Synthetic
Next, we leverage a strong instruction- folloiiinnnooooiinnggg.o model, GPT-4o, as judge to classify which in- struction was synthetic. Given 100 pairs of human-labeled and synthetic instructions, the classifica- tion accuracy ranged from 50% (random guessing) to 68%, with the lowest accuracy for HumanEval and MMLU, suggesting that synthetic data closely mimics human complexity and naturalness in these domains. Conversely, the higher accuracy for BBH and GSM8K indicates that synthetic data in these domains has room to improve.
Finally, we compare the performance of Self-MoE fine-tuned with synthetic data against human- labeled seed data. We observe that with the same quantity (400) the seed, synthetic data achieves performance similar to human-labeled data on average. When scaling up the size (5x and 50x), synthetic data demonstrates effectiveness and scalability.
4.9 DISCUSSION ON THE OVERHEAD OF SELF-MOE
One possible concern in adapting LLMs into compositional systems using Self-MoE the potential introduction of overhead. Here, we discuss this aspect in detail, emphasizing that the additional overhead of Self-MoE. is minimal while yielding significant performance gains. Essentially, the expert modules in Self-MoE are lightweight LoRA modules, contributing only about 1 1% additional parameters (total) for four experts, as detailed Table (Total Params). These experts are sparsely activated, which maintains low active parameters (7B + 0.3%) during inference, thus efficiently minimizing inference overhead. In contrast, traditional MoE models like Mixtral (Jiang et al., (024) and BTX (Sukhbaatar al. 2024) typically employ a feedforward network (FFN) layer for each expert, resulting in a significant proportional increase in total parameters as the number of experts grows, indicated in Table 7 which demands much more memory for model loading. The design

---

Published as a conference paper at ICLR 2025
choice in Self-MoE leads to better scalability and resource efficiency, especially when the number of experts is scaled to incorporate numerous domains of expertise.
RELATED WORK
Combination of Experts. There have been numerous efforts to combine the strengths of multiple models modules. The Mixture of Experts (MoE) models such as Switch Transformer (Fedus) et al. 2022), GLAM (Du et al. 2022), and Mixtral (Jiang et al. 2024) exemplify this, dynamically allocating tasks based on the expertise of each component for better efficiency and scalability. These models contrast with ours by not prioritizing lightweight experts, resulting in larger model with more parameters. Unlike their experts implicitly learned during pre-training, Self-MoE explicitly creates semantic experts for targeted improvements.
Another relevant area is merging, involving the weighted averaging of multiple models form single, aggregated model (Wortsman et al. 2022 Matena & Raffel 2022 IIharco et al.) 2023 Jin
jtt al. 2023). One of the leading methods, TIES Yadav al. (0233 tackles conflicts and parameter inconsistencies among models. DARE (Yu et al. (2024] further reduces the redundancy of param- eters. However, these methods are fundamentally static in that they operate with fixed parameters once merged, which may lead to interference, lacking the dynamic flexibility that MiXSE offers.
There exist notable recent MoE models that similarly explore the utilization of semantic experts, al- beit in distinct contexts (Gururangan al. 2022 Wu et al. 2224 Muggeth et al.) 2024 Suukhbattar et al. 2024). MOLE relies on homann-lablld data, and PHATGOOSE assumes the availability of existing expert models trained by external creators and necessitates additional training for router on the creators' side. DEMix and BTX rely on extensive pre-training, demanding significant resources, yet as a pre-trained model holds the potential complement our self ceff-training approach. Un- like MOLE and PHATGOOSE, our Self-MoE framework creates experts and a router from scratch through self-improvement, while using minimal resources, as contrasted to DEMix and BTX. To offer broader perspective, Table 7 in Appendix presents comprehensive summary of various models that, while relevant, are not directly comparable. For further discussions and more detailed comparison, please refer to Appendix D.I
Self-Improvement and Specialization of LLMs. LLMs often revolves around an Instruction-tuning scheme, which can significantly boost cross- task generalizability (Ouyang et al.) 2022, Su et al.) 2022; Mishra et al. 2022] Wei et al. 2022). Due to the bottlenecks of expensive annotation costs which lead to limited scalability, the self- training concept (Luo 2022) has gained attention from the community, where LLMs are aligned with automatically self-generated synthetic instructions (Wang et al.] 2023 Sun et al. 2023 Li ct al. 2024b). These are distinguished from distillation techniques (Hinton et al. 2015 Kang et al.
The pursuit of enhancing the capabilities of
2023
which assume stronger teacher model (Mitra al. 2023 Li et al. 2024a Sudalairaj al. limiting their applicability.
2024
With the growing need to adapt generalist models specific domains, Kang et al. (2024) adopts the self-training for specialization, tackling that general instruction tuning is rarely effective in ex- pert domains. While this work lays a foundation for enhancing specialized expertise with minimal resources, we recognize inherent trade-off's in a monolithic structure, such as performance compro- mises outside specialized domains. Conversely, our Self-MoE achieves uncompromising multiple expertise with a modular approach without extensive resources and adding many parameters.
CONCLUSION
In this study, we proposed Self-MoE to build compositional LLMs with self-specialized experts, MiXSE, to enhance targeted capabilities, adaptability, and interpretability without the reliance on ex- tensive human-labeled data. Empirical evaluations across diverse domains with multiple base mod- els demonstrated that MiXSE significantly enhances base LLM performance and overcomes spe- cialization trade-offs. We believe this work offers a step towards modular, self-improving paradigms which can address the inherent limitations of monolithic models, providing a promising direction for future L.M research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research is supported in part by the NSF under grant number IIS-2052498. Any opinions, findings, and conclusions or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
@I@Mett Meta. Llama 3 model card. 2024.
URL h.tps://gittub.ccmm//meet Llama/
11am33/bobbmainn/ /MMDEL CARD
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021.
Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. A framework for the evaluation of code generation models. https: /tttts::/gttie //g/tttubbs com/ harnnss 2022.
bigcode--rroject//bbgee..
eva
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432-7445 April 2020. ISSN 2159-5399. doi: 10.1 11600//aai.v34405.66..
URL http::///xxxdii.///100000 11t.:/1...619//A00 /AAII 034I05.6222
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford. Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., neurips cc ccpppprr wwwess/pppee///22// 2020. URL File///5cccbrro0551111ssssslee11 Papeee.
Eric L. Buehler and Markus J. Buehler. X-lora: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular de- sign, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Daixuan Cheng, Shaohan Huang, and Furu Wei. Adapting large language models via reading com- prehension. In The Twelfth International Conference Learning Representations, 2024. URL
net. /ttp:
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson. Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent

---

Published as a conference paper at ICLR 2025
Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1-53, 2024. URL http:///mm httt:://mmrr.rrgppaapp..
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient finetuning of quantized LLMs. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL https:://oeereerre
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Heellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le. Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569. PMLR, 17-23 Jul 2022. URL https::/prrceeete
www.............m.. v1 du22c.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(10)):1 2022. URL / jmlr. org/ http://aaasss// htt.ss///3/21111 w
html
Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling LLMs' knowledge gaps with plug-in specialized language models. In The Twelfth International Conference Learning Representations, 2024. URL https:
hww.:nddwwwwwxoii.. //openreview. oott:/oorum..d-/boto.......
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL 10256836
org/ records/
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. DEMix lay- ers: Disentangling domains for modular language modeling. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5557-5576, Seattle. United States, July 2022. Association for Computational Linguistics. doi: 10. 106118663vvll 122022 mawcc-maii..000 URL https:://allanthlllle. orr/ 2022. www.aaaclllmac matacc-mmain 407
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021a. URL htttts:://ppppreeeeste www.:/.eeerfrrm/m// forum?ddddd
net
37KBimI 3GmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL patpp:// //arxiv. org/abs 1503. 02531
12

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL hattps://operreyes htt.:eww.eeetttteeeeeeeeeeee forum?
Tid=NZeVKeeFyf)
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2023.
Jianheng Huang, Leyang Cui, Ante Wang. Chengyi Yang. Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self- self-synthesized rehearsal, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httptttp//// http///pppereii..e// net www.:nerttootroeo. forum?id GtCKkf8 jrj
/
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang. Timothée Lacroix, and William Sayed. Mistral 7b, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gi- anna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Piete Piettoo and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. htRthttss: /ttps::////p/r////////ee//
n forum?16=FTCohuRR5
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi:
URL /ttpps:/aaclattaaae
org/
P1 P17-1147
Junmo Kang, Wei Xu, and Alan Ritter. Distill or annotate? cost-efficient fine-tuning of com- pact models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 11100-11119, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18653//1/2023.add lon- URL https: http::://aalaaathlllogo org//023.
aCl ong. 622
Junmo Kang. Hongyin Luo, Yada Zhu, Jacob Hansen, James Glass, David Cox, Alan Ritter, Rogerio Feris, and Leonid Karlinsky. Self-specialization:: Uncovering latent expertise within large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Find- ings of the Association for Compututional Linguistics ACL 2024, pp. 2681-2706, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: pettss//aclanthee
.rg. 2024. wiwdings-acc..1......
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-experts from dense checkpoints. The Eleventh International Conference on Learn- ing Representations, 2023. htt opwenevveeeeeeeeeeeeeeeeeee
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forget- ting in language models via implicit inference. In The Twelfth International Conference on Learn- ing Representations, 2024. URL /ttpa/////////////e/ee//ee httprr/rweeeeeeeeoooooooo/oo/eae
13

---

Published as a conference paper at ICLR 2025
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le. and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, 2019. doi: 10.1 Lapttalla__0000 00276. URL
attts:://aclannholooyy -twp:ttcoiooy..or/01--111000
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang. Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang. Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024a.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-train-mergg:: Embarrassingly parallel training of expert language models, 2022.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. In The Twelfth International Con- ference on Learning Representations, 2024b. URL /ttps:://penrevvev net forum?
id=loi (HJRRTT
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, I Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khat- tab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of lan- guage models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: bettps://epenreviee net foorumidd=104Lziba Certification.
Featured Certification, Expert
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguis- URL https: htt///ccaanttooogoo/o org/2022.
tics. doi: 1/18666//12222222/-loooo
acl-iong, 229
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis. Luke Zettlemoyer, and Wen tau Yih. RA-DIT: Retrieval-augmenner dual instruction mning. In The Twelfth International Confer- ence on Learning Representations, 2024. URL bttps:://ppezeee/// 220Tbutug9
net/ / forum?id-
Chen Ling, Xujiang Zhao. Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, Tianjiao Zhao, Amit Panalkar, Dhagash Mehta, Stefano Pasquali, Wei Cheng, Haoyu Wang. Yanchi Liu, Zhengzhang Chen, Haifeng Chen, Chris White, Quanquan Gu, Jian Pei, Carl Yang, and Liang Zhao. Domain specialization as the key to make large language models disruptive: A comprehensive survey, 2023.
Hongyin Luo. Self-training for natural language processing. Ph.D. thesis, Massachusetts Institute of Technology, 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: Sfatc-of-the--a the parameter-efficient fine-tuning methods. https: //thhb... 2022.
14

---

Published as a conference paper at ICLR 2025
Michael S Matena and Colin A Raffel. Merging models with fisher- fisher-weighted averaging. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 17703-17716 Curran Associates, Inc., 2022. URL. btttps: / /ttp::pee/pppoopeeedeeeeo ngs. neurips ct.ppper.ppppp.eeepeeeeeee httpsscccpppeeerieseeeeeepe//eee file File/70c26930 wfw.06226ffff3444400020
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. MetalCL: Learning to learn in context. In Marine Carpuat, Marie-Cathert de Marneffe. and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2791-2809, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.118553/112220 /2t/222222222222 main. 201.
org 2022 wtwp:///o22....aala....a nwacl-alaaaaa ma..n.2...... 201
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACL, 2022.
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agar- wal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. Orca 2: Teaching small lan- guage models how to reason, 2023.
Mohammed Mugeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among special- ized experts for zero-shot generalization. 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35; 27730-27744, 2022.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLP-IJCNLP),, pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/D11-14110 [ttt:://s:///al anthhoanntooo wanthologyy.oom org.. 90/e:/199-1119
Hongjin Su, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, Tao Yu, et al. One embedder, any task: instructioo-finetuuedd text embeddings. arXiv preprint arXiv:2212.097411 2022.
Shivehander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots, 2024.
Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozière, Jacob Kahn, Daniel Li, Wen tau Yih, Jason Weston, and Xian Li. Branch-train-mix: Mixing expert Ilms into mixture-of-experts Ilm, 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. In Advances in Neural Information Processing Systems, 2023.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou and Jason Wei. Challenging big- bench tasks and whether thain-of-thought can solve them. arXiv preprint arXiv:22 10.09261, 2022.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing 1lama model. https: //gtthb///tubb////// com/ wwweeoottsstte ttttttttstaaa 1ab/ sttsssaaooorr.ooooooooooooooosos alpaca] 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex
15

---

Published as a conference paper at ICLR 2025
Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, An- tomia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Chune -hho, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Hen- ryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff' Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel I. Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg. Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet. Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis. Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, Mosharaf Chowdhury, and Mi Zhang. Efficient large language models: A survey. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Survey Certification.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 13484- 13508. Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/ www.233.cllooo..55... URL wttpt::/p////t. /at/aaaaaannncccocaeeaaaaaaa orq 2023 www.:23.11111....
754
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations, 2022. URL
net/
Forum?.d==FErGGozddd
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Lud- wig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accu- racy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 29965. PMLR, 17-23 Jul 2022. URL https: htttp:://proceedingge .mlr. wwt.m.rr.ees//e.em.. v162 62/62/22 2/2//ortsss. sman22a. html
Hongqiu Wu, Linfeng Liu, Hai Zhao, and Min Zhang. Empower nested Boolean logic via self- supervised curriculum learning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13731-
16

---

Published as a conference paper at ICLR 2025
13742, Singapore, December 2023. Association for L Linguistics. doi: 10.18653/v1/ 2023.emnlp main.847. URL httppp/:///llaantoooooo........ org/ 000.0233..mmmmmmmmamm 847
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of loRA experts, In The Twelfth International Conference Learning Representations, 2024. URULhtpss//ppenreeieeww.t
idduuvvBBCh44
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TeEg-merging: Re- solving interference when merging models. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models super mario: Absorbing abilities from homologous models as a free lunch. In International Conference on Machine Learning. PMLR, 2024.
Matei Zaharia, Omar Khattab, Lingjiao Chen, Jared Quincy Davis, Heather Miller, Chris Potts, James Zou, Michael Carbin, Jonathan Frankle, Naveen Rao, and Ali Ghodsi. The shift from models to compound ai systems. htttps://bats berkeley. edb.blogg/224/02/111 compound-ate ai- systems/ 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? Anna Korhonen, David Traum, and Lluís Märquez (eds.). Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.
18653/v1/P19-447. URL https: http://:///cllaanooooooooooooo wttphooooooyyorr/11111717
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets LLM finetun- ing: The effect of data, model and finetuning method. In The Twelfth International Confer- ence on Learning Representations, 2024. URL
net forum?id=
5HCnKDeTws]
Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. Explainability for large language models: A survey, arXiv preprint 2xXi:23009.0129, 2023.
17

---

Published as a conference paper at ICLR 2025
A EXPERIMENT DETAILS
We provide each of our self self-sppcializatiion prompis for knowledge, reasoning, math, and coding experts in Tables II 12 13 and [4 We largely follow Kang et al. (2024)'s prompt structure to ensure quality, with additional doman-specific instructions that inform task-related information.
For our evaluation, we employ popular and widely accepted evaluation frameworks to pursue stan- dard evaluation setups and protocols: HELM (Liang al.] (023), LM Evaluation Harness (Gao et al. 2023), and BigCode Evaluation Haresss (Ben Allal et al., 2022). We use Huggingface PEFT
(Mangrulkar et al. 2022) and XLoRA (Buehler & Buehler 2024) for the implementation of MoE
compatible with LORA.
Regarding seed instructions, we sampled 100 training instances from each of the MMLU, BBH, and GSM8K datasets, for knowledge, reasoning, and math domains, respectively. For coding, since the size of the HumanEyal dataset is very small and thus the training set not available, we took 100 samples from the MBPP training set and converted the task format to make them suit the Hu- manEval.
During instruction generation, we use three seed data, which are randomly sampled, as in-context examples, using a temperature of and top-p of 0.98, whereas we use five seed data in-context for response generation with greedy decoding. For specialization, we use LoRA applied to all modules with a rank of and alpha 16, and train it using a learning rate of 3e-4, epochs of 3, and batch size of 32. We train each module and MiXSE using a standard Alpaca (Taori et al. 2023) prompt template on a single A100-80GB, which takes only a few hours.
B LIMITATIONS
While our study demonstrates promising results for the Self-MoE, we recognize areas requiring further investigation in future work. Employing self-specialization Kang et al. (2024) to generate synthetic data within our framework may raise concerns about potential data contamination and noise. Nonetheless, findings from Kang et al. (2024), which conducted an n-gram overlap analysis between the speciapecialiiooionn data and test data, confirmed no significant overlap, thus alleviating the concerns about contamination. Despite this, the need for continuous monitoring of potential bi- ases from pre-training and the development of enhanced data validation and noise filtering strategies remain important, and may present interesting direction for future work. Moreover, due to compu- tational constraints, we did not scale our model and data to their full potential. We also did not work on the optimization of the XLoRA, the MoE module used, to focus purely the research prob- lem defined in this study. Future work should therefore concentrate on overcoming these limitations, which will enable better data quality and more extensive training unveil the full potential of the Self-MoE framework.
Table 6: Dataset statistics. Non- Kon-Target (In-Expertise) indicates where MiXSE does not directly specialize using seed data directly while relevant to targets. Non-Target (fut-of-Expertiic refers to irrelevant cases.
Category
Benchmark
# Examples
Torget
Academic Knowlsdge
MMLU (57 Tasks)
14,079
Reasoning
RBH (27 Tasks)
6,511
Math
GSMSK
8,790
Coding
HumanEval
164
Non Target (In- Expertiiss)
Math
MATH
12,500
Coding
MBPP
257
Non Torget (Out- of Expertixe)
World Knowledge
Natural Questions
3.610
TriviaQA
17,200
Hellaswag
10,000
Commronnsense
PIQA
Safety
TruthfuQQA
817
18

---

Published as a conference paper at ICLR 2025
Table 7: Additional comparisons with other models for references. Results extracted from each corresponding paper, except for pre-training methods where the numbers are all from BTX (Sukhbattar et 2024).
Meiked
Param
Param
libeal
lispens
viight
Labele
(MNLLA) S-dee)
(MBI))
Garee 70 a 2001
18
18
66-7
56.1
43.5
36.1
MI
631
BoadeeTraa Mage (66 Lesal
<343
11J8
44.5
21-2
3MA
Noath Thae (1676)
1R
$3.5
$3.1
76.7
<40
23B
GLANTR TI G Lue. PAREs
1B
18
62.
607
R0A
48.5
Matesk:7B (w/.////iaaaaaa www.Kaatrrnn Scarel 2004
18
649
66. 611 535
C DATASET DESCRIPTIONS
The statistics for each dataset are provided in Table The target datasets used are as follows:
MMLU (Massive Multitask Language Understanding) (Hendrycks et al. 202 : A collection of
57 academic knowledge tasks.
BBH (BIG-Bench Hard (Suzgun al. 2022): A set of 27 challenging reasoning tasks.
GSM8K (Grade School Math 8K) (Cobbe et al. 2021): A diverse set of grade school math word problems.
HumanEval (Chen et al. 2021): A hand-written evaluation set for python programming prob-
lems.
D ADDITIONAL RESULTS
D.1 ADDITIONAL COMPARISON AND DISCUSSION
In T present additional comparisons with various other models and methods to provide broader perspective, though comparisons may not appear to direct, due to factors involved such parameters, resources, etc. We diseuss
noteworthy points.
Notably, although MiXSE significantly improves upon its base model, Gemma 7B, it does not yet reach the performance levels of the more powerful Mixtral 8x7B. It is important to understand that Mixtral also utilizes an MoE (Mixture of Experts) architecture, but unlike MiXSE, it does not prioritize lightweight experts, leading Moreover, while Mixtral's experts are implicitly built during pre-training, MiXSE explicitly creates semantic experts, allowing for targeted improvements and clearer interpretability, Importantly, our self-improving method can be potentially applied on top of any pre-trained model including Mixtral in principle.
much larger model with significantly more parameters.
Similarly, BTX (Branch- (Branch-Train-Me Train-MiX) uses pre- te-trriinin MoE strategy where parameter-heays heavy se- mantic experts are employed, yielding substantial enhancements over the base LLM. This approach highlights the effectiveness of using semantically rich experts to refine the model's capabilities. To make comparisons terms of efficiency, our model uses fewer parameters (7B), compared to BTX (12B active with much more whole parameters) and requires only about GPU day for training, compared to 900 GPU days for BTX. In essence, since BTX is also pre-training method while spe- evidenced in previous work (Kang
cialized, we expect it to be complementary et al. 2024).
With shared spirit, MOLE and PHATGOOSE build MoE (Mixture of Experts) using LoRA, which is semantic and lightweight. However, there are significant differences in foundational as- sumptions: MOLE depends human-labeled data, while PHATGOOSE requires access to pre-
19

---

Published as a conference paper at ICLR 2025
Table 8: Detailed results of Self-MoEs w/ other LLMs, comparing with each corresponding LLM and instance merging top of it. For MMLU, we employ the O-shot setting, based on established observations (Dettmers et al. 2023; Lin et al. 2024) that tuning yields only marginal effects in the 5-shot setting for this task. Notably, see that any tunings improve MMLU yet still, our MiXSE demonstrates noticeable average gains over instance merging for most base models.
Knowledge (MMLUU
Reasoning (BBII)
Math (GSMSK)
Coding (HumanEval)
Method
Avg.
3 8B
Base LLM
31.6
60.8
49.0
26.2
41.9
Instance Merging MiXSE
62.5
46.9
47.5
24.4
45.3
61.7
61.5
52.0
29.3
51.1
Genma 7B
Base LLM Instance Merging MiXSE
58.4
42.5
34.1
47.8
62.6
57.6
53.5
36,0
52.4
65.6
61.1
52.5
37.8
54.3
LLMMA-2 7B
Base LLM Instance Merging MiXSE
17.8
38.5
13.0
12.8
20.5
45.2
36.8
13.0
13.4
27.1
44.0
38.3
13.5
14.0
27.5
LLMMA-2 13B
Base LLM Instance Merging
20.4
45.6
22.5
16.5
51.2
43.0
25.5
17.1
34.2
MiXSE
52.1
45.6
25.0
17.1
35.0
Mistral 7B
Base LLM
29.8
54.9
38.0
27.4
37.5
Instance Merging
61.7
51.5
30.5
29.2
43.2
MiXSE
62.0
58.1
38.0
28.0
46.5
trained expert models developed externally. In contrast, our Self-MoE framework independently constructs both experts and router entirely from scratch, focusing on self-improvene without such dependencies. While their scenarios are considered reasonable certain context, we aim for broader applicability by minimizing assumptions on conditions.
Lastly, GLAN demonstrates outstanding performance across various domains. This is attributed to their reliance distilling from the larger and stronger model, GPT-4, using a huge amount of data (e.g., million). As outlined in our problem statement (Section 2}. we deliberately avoid assuming the availability of such advanced models to ensure the broader applicability of our method which self-improves from scratch. Consequently, while acknowledging each their own value, is crucial to recognize that direct comparisons may not be entirely appropriate, given the fundamental differences resource assumptions and initial conditions.
D.2 DETAILED RESULTS OF SELF-MOE WITH OTHER BASE LLMS
Table g presents the detailed results of our Self-MoE applied to a diverse set of base LLMs includ- ing LLaMA-3 8B, Gemma 7B, LLaMA-2 7B and 13B, Mistral 7B, As discussed in 4.5 overall, our approach can improve base models, outperforming the strong instance merging bascline, particu- larly with newer/stronger models like Gemma 7B, Mistral 7B, and LLAMA-3 8B. specific cases like LLaMA-2 for reasoning. however, we see no improvement, while improving on average. This can be attributed to the weaker baseline performance, which hinders the generation high-quality specialized synthetic data for specific capabilities. Through manual inspection of small sample sets, we identified instances where the generated instructions exhibited poor quality, including issues such repeated tokens, off-topic content, and other inconsistencies, not following given instruc- tions. This highlights an opportunity for further refinement in synthetic data generation techniques, which view complementary area of ongoing research. As methods for synthetic data genera-
20

---

Published as a conference paper at ICLR 2025
Table 9: Results of MiXSE using only seed data. Seed Only training shows only marginal improvements over the Base LLM in some benchmarks, validating that the effect of Self-MoE is not merely due the use of seed data.
Benchmark Base LLM
Seed Only
MiXSE
Knowledge
58.3
57.4
65.6
(MMLU)
Reasoning
57.0
(BBH)
Math
42.5
45.0
52.5
(GSM8K)
Coding
34.1
34.1
37.8
(HumanEvvl)
Avg.
47.8
48.4
54.3
Table 10: Visualized examples of the token-level routing, where each token highlighted according math coding Different experts can be dynam-
to assigned experts knowledge, reasoning ically activated within an instance, as the routing operates at token-level, while the most relevant expert is primarily selected.
GSMBK
Q: Shawnda decides that neighborhood kids could really use a bike inflution service.
She decides the
best way
charge by
Ube
tire. Each tire costs cents to inflate.
On the trieycle tires intlated. Finally, one person
first day,
people
bicycles came
by
get both tires inflated.
people came by to get all their
actually came by on
unicycle.
How dolllars did she make
that day?
A: people an
bicycles came by to get both tires inflated.
So that
is
10 tires.
3 people get their
tricycle tires inflated.
So that
lires.
F'inally, one person
uctually cume by on
unicycle. that
tire.
So total, there
10
9
20
tires. Since each tire costs 25 ocats, she made 20 00.25 =
dollars.
The answer
HumanEval
from typing import List
def clemeuts(mumbers:: List[float], threshold: float)
Cheak if in given list numbers, are any
tWD
numbees closer to each other than given threshold.
}>) has.close.elements
Falsc
}> hac_close_elmmenss
1.0
2.8
3.0
4.0
5.0
2.0 1.
033
for rangeclentoumbers yx
For in range(i
len/numbers
abs(numbersjil numbersfjD) threshold:
retura
True
return
False
21

---

Published as a conference paper at ICLR 2025
tion advance, they can directly enhance Self-MoE's performance with better self-specialized expert modules.
D.3 MIXSE USING ONLY SEED DATA
Table 9 shows the results of the MiXSE when exploiting only seed data for training, clarifying the benefits derived from our methodological enhancements beyond the mere inclusion of seed data in training. While the Seed Only shows slight improvements over the Base LLM some bench- marks, the significant enhancements of our MiXSE across all benchmarks confirm that the enhanced capabilities of Self-MoF. are not merely due the use of seed data. This further highlights the achievement of iefoovmprveeentt with our method.
D.4 VAILDITY OF COMPARATIVE RESULTS
In an effort to address the concern related to the sensitivity of in-context learning (Min et al. 2022). we perform three runs with the different lists of few-shot samples where applicable. As result, we see that the mean of the base LLM (Gemma-7B)3 average performance across domains is 47.9 with standard deviation (SD) of 0.56, that of our MiXSE is 53.6 with an SD of 0.60, and that of instance merging is 51.6 with an SD of 0.87. A statistical analysis between MiXSE and instance merging yields p-value of 0.03, confirming the significant difference.
D.5 VISUALIZED EXAMPLES OF ROUTING DECISION
Table T0 provides detailed visualization of token-level routing decisions based the Top-1 lection configuration. This table highlights how the routing module dynamically activates different experts within single instance, reflecting the flexibility of token-level operation. As illustrated, the most relevant expert is predominantly selected for each token; however, the system occasionally ac- tivates other experts dynamically, depending on the specific token context within the instance. This behavior contrasts with speiaapecili,,tion,, which consistently relies on single expert to handle all tokens uniformly, lacking the token-level granularity observed the routing mechanism.
22

---

Published as a conference paper at ICLR 2025
Table 11: Prompts for knowledge-related instruction and response generation.
Instruction Brainstorming Prompt
You ace aaked Lo gome wilh set of Lask austructiom about diverse domsins aurose STEM, humanities, social sciences, and ethers. These cask instructions given language model and we will evaluate the model for completing the instractions.
Here are the requirements:
type task should be nultiple-choice question answering. That is, question aleng with miltipla options (A, B, C, D) should be proviced.
2, The langusge used for the rnst ton/questioo also shoold diverse.
3. A language model should be able to complete the instruction. For example, do not ask the assistant to create any viaual or audio outpit. For another example, not ask the asalatant
Lo wake you at Spm sel reminder becaure it cannet perforn any action. The ractrocions shoold be in English. sentences long. Bither imperative sentence
5, The instruction lons should be
question is permitted.
You should generate appropriate inpul Lo the instruction. The input field should contain specifiz exemple provided for the instruction. It should invelve realistic data and should contain simple placeliclders. The inpul should provide substantind conteal to make the instruction chellenging.
covered for extensive expest-level knwwledgs. may include Abstract Algebra, Anatomy, Astroncmy, Business Btaics, Clinical Knewledge, colloge-level Riology, Chemistry, Computer science, Mathematics, Medicine, Physics, Computer Security, Conseptusl Physics, Foonometrias, Electrical Engineering, Elamentary Mathemation Formal Logic, Gioosl Facts, High School-level Biolcgy, Chemistry, Computer Science, scropean statory, Geogrophy, Gou't and Politics, Macroeconomt es, Mathematica, Microeconomi es, Phyaics, Psychology, Statistics, 0S Hietory, Norld History, taw, duriaaprudene,, Togioal Fal Paliacies, Machine learning, Managemenz, Marketing, Medical Genetics, Miscellaneous, Moral Disputes, Professi moalnnaleeee !Accoun;ing lax, Kediaine, Paychol ogy)), Piblic Relations, Security Studies, Seciology, Foreign Policy, Virology, Korld Religione,
Ensure civerse domains
subjects
Aging,
Sexuality, Internationa:
Scenarios, Nutrition, Philosophy, Prehistory,
List
Response Generation
You are know edgesble domain expert. Gliven an instruction and queation, generate the best answer Lo solve the given Lask about 8T3M, humanities, social sciences, and others.
Table 12: Prompts for reasoning-related instruction and response generation.
Instruction Brainstorming Prompt
You are asked to come with a set of task instructions focusing on challenging tasks that require multi-step reasoning. thess task instructions will given to language model and we will evaluste the model for completing the instructions.
Here are the requizements:
The type task should be question anavering, raquiring milti-stap reasoning. ion/qustionn also should be diverse.
2. The language used for the
3. The generated problem should have single correct answer.
4. The inst fons should be 'n Fnglish.
The instructions should be to 2 sentences long. Bither an: imperative sealence or question perritted,
6. You should generate an apprupriate input question to the instruction. It should
involue realistic data and ahou'd d not contain simple placeholders. The input should provide subsiantial content to make the inst ion challenging.
covered for extensive expert-level reasoning. The Lasks may be about boolean expression, causel judgement, date understanding, disembiquation of question, closing Dyck-n words, formal fallscies, geometric shapes, hyperbaton, logical deduction of abjects, movie racommanaation, milti-stap arithmetic problem, navigation, object counting, table ressening, reasoning about colored objects, selecting that cuins the in an input, salient translation ervor detection, sarcastic sentence classitication, sports understanding, temporal sequences, tracking shuffled objects, 11ea, word sorting, etc.
1. Insure diverse topics and levels
Mat tasks:
Response Generation
You are multi-stap reasoning expert. Given an instruction and challenging question, generate step-by-step reasoning and the answer.
23

---

Published as a conference paper at ICLR 2025
Table 13: Prompts for math-related instruction and response generation.
Instruction Brainstorming Prompt
instructions focusing en machematical problems. These taak instructions will be givan to language model and we will evaluate the model nor completing the instructions.
come up
Fere are the requitements:
The of Lask should be question answering, requicing mulli-atep ressuning,
language used for the rnst fon/question also shoold diverse.
problem should have solution 1. The instructions should be in English. sentences long. Bither imperative seatence
The genersted methematico
The instructions Lons should be question is permitted.
question the instruction. It should involve realistic data and should not contein simple placeholders. The input should provide substantial
You should yenerale appropriate
make 1. Insure diverse topics and Levels are covered for extensive expert-level reasoning, The subjects may include Algebra, Counting, Probability, talemlus, stazistics, Geometry, Linear Algebra, Number Theory and grade school math, etc.
instruction chailenging.
Mist of tasks:
Response Generation
math expart. instruction mathematical question, generate
step-by-step reasoning and the answer.
Table 14: Prompts for coding relate- instruction and response generation.
Instruction Brainstorming Prompt
instructions focusing on coding problens. These task instructions will be given to language model and we will evaluate the model completing the instructions.
aaked come up set
Hare are the requirements:
1. The of task should be about codingg problems, auch sa writing python funet fon given
specific instrustion and Lest examples.
2. The language used for the inst fon should be diverse, but the programming language
should be pythun,
3. The genersted problem ahould have solution.
1. The instructionn ions should be in English.
You sheuld
test examples for the problen. Insure diverse funetions and levels are covered for extensive expert-leved -evell coding.
apprepriate and
List of casks:
Reaponse Generation
coding expert. instruction tast cases, wrize python
passes the Lest cases.
24


Reviews:
[{"review_id": "WtIV0zrmoc", "reviewer": "Reviewer_tfML", "summary": "This paper presents a novel approach to modularize large language models by constructing expert modules from self-generated synthetic data and creating a compositional system. Unlike previous Mixture of Experts approaches that use LoRA and rely on either human-labeled data or pre-trained modules, this method develops modules from scratch, adapting the model to specific tasks. Experimental results demonstrate the advantages of this approach over base LLMs and self-specialized LLMs in multiple tasks.", "questions": "- Router Details: The router is crucial to the Self-MoE model’s performance but is described only at a high level in the paper. More detailed information (e.g., training etc.) in the main body would enhance the clarity and accessibility of the approach.\n\n- Synergies Across Domains: One reason for the Self-MoE’s superior performance might be that LLMs can exploit synergies across areas of expertise. Would this advantage persist if the domains were highly distinct, such as medical vs. finance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Mitigation of Forgetting: Compared to monolithic models, which often face challenges with knowledge retention, the proposed approach maintains the integrity of each expert module, which enhances the overall model's performance and adaptability.\n\n- Lightweight and Synthetic Data-Driven Modules: The method constructs individual, lightweight expert modules from synthetic data, which bypasses the need for human-labeled data and broadens the scope of applicability.\n\n- Generalization: The generalization tests indicate that the Self-MoE approach offers benefits beyond the targeted tasks, improving performance on benchmarks that were not explicitly used in training.", "weaknesses": "- Limited Model Sizes: The experiments primarily focus on small-scale LLMs (7B/13B models), leaving open the question of whether these findings can extend to larger models. Testing on a wider range of model sizes would strengthen the claims.\n- Quality and Diversity of Synthetic Data: How does the approach ensure the correctness, diversity, and quality of the instruction-response data generated for training the expert modules?\n- Domain Granularity: How is the domain granularity determined? For instance, broad domains like reasoning, math, and coding can be subdivided further (e.g., reasoning into medical, finance). To what extent do the findings depend on the chosen level of domain specificity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hw1pgF97LP", "reviewer": "Reviewer_1Lag", "summary": "Self-MoE constructs the corresponding experts modules to realize the self-specialization mechanism to improve the performances on non-specialized tasks and diverse benchmarks such as knowledge, reasoning, math, and coding.", "questions": "1. How does the number of experts impact the downstream performance? If the knowledge for differenct experts is similar, is it possible to reduce the number of experts? \n2. How does the model generate to the unseen tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Introducing the computation overhead realizes better performance improvement\n2. Using the self-optimized routing activates the distinct domain-specific capabilities to help improve the performance shared base LLM \n3. The presentation is good and the experiments are detailed", "weaknesses": "1. The comparison with related works such as  [1] is not enough. The concept of lora-moe has been introduced in this work.  This slightly affects the novelty of the work.\n\n[1] Zadouri T, Üstün A, Ahmadian A, et al. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning[J]. arXiv preprint arXiv:2309.05444, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G9TPaaB5gT", "reviewer": "Reviewer_eqDV", "summary": "This paper proposes a new paradigm of self-MOE that designs a compositional LLM integrating various experts specialized in different aspects and employing a trained router to dynamically choose experts during inference. The experts are trained with LoRA and only a small proportion of parameters are adjusted during self-specialization, keeping a high efficiency. Self-MOE has been tested on several popular LLMs, e.g., Gemma and LLaMA 3, proving its strong ability to enhance current LLMs' performance when dealing with tasks in various aspects simultaneously.", "questions": "1. Line 223. I am just wondering if the top-k operation is after doing softmax, then the probability of expert selection is not normalized, meaning that when k is small alpha will also be small. Do you think it will lead to the experimental results which show that top-1 self-MOE is better than top-2 self-MOE (as presented in Table 2). I think by not normalizing the selection probability, the amount of information introduced by experts will be different when different numbers of experts are activated. So directly comparing top-1 and top-2 self-MOE is a bit unfair. Maybe the reason for the worse performance of top-2 self-MOE comes from the excessive redundant information introduced by an irrelavant expert. But this guess contradicts your claim in line 88 that MiXSE explores synergies among experts. I wish to have a more detailed discussion about this part. Given the hypothesis that synergies among experts help to boost performance, why top-1 self-MOE outperforms top-2 self-MOE? This is important for this work, because by only choosing one expert, it largely resembles self-specialization.\n2. Line 348. I am also suspicious about the claim that jointly training router and experts would make semantic distinctions among experts diluted. Have you tried to jointly train them and activate more experts (>= 2)? Could you provide it in your rebuttal (sorry I know it requires lots of computational resources)? I just think the experiments and the obervations are not fully supported by your claim. This is also related to your claim that synergies among experts happen when you have multiple experts controlled by a router in self-MOE. If you don’t jointly train them, how synergies happen?\n3. Again in Section 4.3, the routing distribution shows that datasets like MMLU benefit from both knowledge and reasoning experts. So why don’t you just activate more experts?\n4. Section 4.7. Have you tried adding experts in different orders? For example, R + M + C + K.\n5. In Table 3, you labeled TruthfulQA as safety. I am not sure why.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel idea of self-MOE and it can be a promising new paradigm for stronger foundation models.\n2. Paper is well-written.", "weaknesses": "1. Some claims are not very intuitive. I have questions about how synergies among expert happen and given the synergies, why only 1 expert is activated. Please refer to the questions below.\n2. Some parts in the main body of this paper are not necessary, for example Table 5. I would rather see it in the appendix. Some figures are either too huge or loosely-structured and leave lots of wasted blank space (e.g., Figure 1 and 2). I think you can put more experiments about self-MOE in the main body. Please refer to point 2 of the questions below. And I would also prefer to see more detailed analysis about other base LLMs. Currenlty I only see a very abbreviated figure (Figure 4) related to this. I think breaking down their performances into several parts as how you present Gemma 7B is meaningful.\n\nI actually like this paper. Please kindly address my concerns and questions. I think a good rebuttal will make me change my mind and raise the score. Good luck with the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVNTv4VeQU", "reviewer": "Reviewer_fc1R", "summary": "This paper proposes a self-specification method, named Self-MoE, to transform a monolithic LLM to a MoE system with self-specific experts. To construct expert modules for specific domains, this paper utilizes self-generated data from the base LLM. The resulted Self-MoE demonstrates substantial improvements over the base LLM, even achieves better performance than data mixture and weight merging.", "questions": "1. In the instruction brainstorming stage, how to generate diverse instructions within a given domain? Since the generator is not a strong instruction-following model, e.g. GPT-4, using self-instruct approach can lead to low diversity in the generated domain-specific datasets as illustrated in [5].\n\n2. In the mixture of self-specialized experts stage, router network parameters are shared across all layers. However, when router networks in different MoE layers do not share parameters, they typically exhibit distinct routing behaviors. What is the influence on overall performance and routing behaviors if separate, layer-specific router networks are employed instead of shared ones?\n\n3. In the main results, the baseline methods involve fine-tuning both the LoRA and router parameters. However, this approach presupposes that all base models should be implemented using the MoE architecture, potentially introducing additional complexities in router training. To ensure a more comprehensive and equitable comparison, it is essential to add additional baseline methods, which are directly fine-tuned on the base model using different datasets (specific capabilities, mixture of capabilities, model merging).\n\n4. In Section 4.5, can the author provide the instance merging results (using all the 20k data to direct fine-tune the base model with LoRA) for different base models?\n\n[5] Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration\n\nI would be happy to discuss further with the authors and reassess my score based on the rebuttal stage.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It's a good idea to stimulate diverse domain-specific competencies inherent within the base model and reassemble these specific experts to achieve a more powerful MoE system. \n2. This approach offers the advantage of eliminating the necessity for domain-specific data acquisition.\n3. The writing and presentation are clear and easy to read.", "weaknesses": "1. The primary contribution of this paper is constructing a MoE system from multiple domain experts, which are fine-tuned on the domain-specific synthesis data from base model itself. However, this self-specialization method was originally proposed by [1]. Moreover, recent studies already propose the construction of MoE system based on multiple domain-specific dense models, which can be fine-tuned using LoRA [2] or full-parameter tuning [3]. \n\n2. A critical question in utilizing self-synthesized data for self-improvement is where the performance improvements comes from. For instance, the improvements in [4] is based on an additional reward model to differentiate between good and bad samples. However, the underlying mechanism of Self-MoE's self-improvement remains ambiguous. It is not immediately apparent why domain-specific datasets generated by the base model, without guidance from a stronger model or reward model, can lead to overall performance gains.\n\n3. The expert modules in Self-MoE are fine-tuned from self-synthesized domain data, which determines the upper bound of the expert's capabilities. Therefore, more analysis of the self-synthesized datasets is needed, e.g., data diversity, complexity, and the model performance compared to using existing open-source datasets.\n\n[1] Self-specialization: Uncovering latent expertise within large language models.\n\n[2] Mixture of lora experts.\n\n[3] Branch-train-mix: Mixing expert llms into a mixture-of-experts llm.\n\n[4] Direct Nash optimization: Teaching language models to self-improve with general preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)