Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts

Authors: Junmo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang, Jacob A Hansen, James R. Glass, David Daniel Cox, Rameswar Panda, Rogerio Feris, Alan Ritter
Primary area: foundation or frontier models, including LLMs
Keywords: Efficient Specialization of LLMs, Self-Improving, Mixture of Experts

Abstract:
We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts, named MiXSE (MiXture of Self-specialized Experts). Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data, each equipping a shared base LLM with distinct domain-specific capabilities, activated via self-optimized routing. This allows for dynamic and capability-specific handling of various target tasks, enhancing overall capabilities, without extensive human-labeled data and added parameters. Our empirical results reveal that specializing LLMs may exhibit potential trade-offs in performances on non-specialized tasks. On the other hand, our Self-MoE demonstrates substantial improvements (6.5%p on average) over the base LLM across diverse benchmarks such as knowledge, reasoning, math, and coding. It also consistently outperforms other methods, including instance merging and weight merging, while offering better flexibility and interpretability by design with semantic experts and routing. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730024903737

thread_type:
official_review

<thread_object>
creator: reviewer_fc1r

<content>
summary:
This paper proposes a self-specification method, named Self-MoE, to transform a monolithic LLM to a MoE system with self-specific experts. To construct expert modules for specific domains, this paper utilizes self-generated data from the base LLM. The resulted Self-MoE demonstrates substantial improvements over the base LLM, even achieves better performance than data mixture and weight merging.

strengths:
1. It's a good idea to stimulate diverse domain-specific competencies inherent within the base model and reassemble these specific experts to achieve a more powerful MoE system. 
2. This approach offers the advantage of eliminating the necessity for domain-specific data acquisition.
3. The writing and presentation are clear and easy to read.

weaknesses:
1. The primary contribution of this paper is constructing a MoE system from multiple domain experts, which are fine-tuned on the domain-specific synthesis data from base model itself. However, this self-specialization method was originally proposed by [1]. Moreover, recent studies already propose the construction of MoE system based on multiple domain-specific dense models, which can be fine-tuned using LoRA [2] or full-parameter tuning [3]. 

2. A critical question in utilizing self-synthesized data for self-improvement is where the performance improvements comes from. For instance, the improvements in [4] is based on an additional reward model to differentiate between good and bad samples. However, the underlying mechanism of Self-MoE's self-improvement remains ambiguous. It is not immediately apparent why domain-specific datasets generated by the base model, without guidance from a stronger model or reward model, can lead to overall performance gains.

3. The expert modules in Self-MoE are fine-tuned from self-synthesized domain data, which determines the upper bound of the expert's capabilities. Therefore, more analysis of the self-synthesized datasets is needed, e.g., data diversity, complexity, and the model performance compared to using existing open-source datasets.

[1] Self-specialization: Uncovering latent expertise within large language models.

[2] Mixture of lora experts.

[3] Branch-train-mix: Mixing expert llms into a mixture-of-experts llm.

[4] Direct Nash optimization: Teaching language models to self-improve with general preferences.

questions:
1. In the instruction brainstorming stage, how to generate diverse instructions within a given domain? Since the generator is not a strong instruction-following model, e.g. GPT-4, using self-instruct approach can lead to low diversity in the generated domain-specific datasets as illustrated in [5].

2. In the mixture of self-specialized experts stage, router network parameters are shared across all layers. However, when router networks in different MoE layers do not share parameters, they typically exhibit distinct routing behaviors. What is the influence on overall performance and routing behaviors if separate, layer-specific router networks are employed instead of shared ones?

3. In the main results, the baseline methods involve fine-tuning both the LoRA and router parameters. However, this approach presupposes that all base models should be implemented using the MoE architecture, potentially introducing additional complexities in router training. To ensure a more comprehensive and equitable comparison, it is essential to add additional baseline methods, which are directly fine-tuned on the base model using different datasets (specific capabilities, mixture of capabilities, model merging).

4. In Section 4.5, can the author provide the instance merging results (using all the 20k data to direct fine-tune the base model with LoRA) for different base models?

[5] Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration

I would be happy to discuss further with the authors and reassess my score based on the rebuttal stage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730572620172

thread_type:
official_review

<thread_object>
creator: reviewer_eqdv

<content>
summary:
This paper proposes a new paradigm of self-MOE that designs a compositional LLM integrating various experts specialized in different aspects and employing a trained router to dynamically choose experts during inference. The experts are trained with LoRA and only a small proportion of parameters are adjusted during self-specialization, keeping a high efficiency. Self-MOE has been tested on several popular LLMs, e.g., Gemma and LLaMA 3, proving its strong ability to enhance current LLMs' performance when dealing with tasks in various aspects simultaneously.

strengths:
1. Novel idea of self-MOE and it can be a promising new paradigm for stronger foundation models.
2. Paper is well-written.

weaknesses:
1. Some claims are not very intuitive. I have questions about how synergies among expert happen and given the synergies, why only 1 expert is activated. Please refer to the questions below.
2. Some parts in the main body of this paper are not necessary, for example Table 5. I would rather see it in the appendix. Some figures are either too huge or loosely-structured and leave lots of wasted blank space (e.g., Figure 1 and 2). I think you can put more experiments about self-MOE in the main body. Please refer to point 2 of the questions below. And I would also prefer to see more detailed analysis about other base LLMs. Currenlty I only see a very abbreviated figure (Figure 4) related to this. I think breaking down their performances into several parts as how you present Gemma 7B is meaningful.

I actually like this paper. Please kindly address my concerns and questions. I think a good rebuttal will make me change my mind and raise the score. Good luck with the rebuttal.

questions:
1. Line 223. I am just wondering if the top-k operation is after doing softmax, then the probability of expert selection is not normalized, meaning that when k is small alpha will also be small. Do you think it will lead to the experimental results which show that top-1 self-MOE is better than top-2 self-MOE (as presented in Table 2). I think by not normalizing the selection probability, the amount of information introduced by experts will be different when different numbers of experts are activated. So directly comparing top-1 and top-2 self-MOE is a bit unfair. Maybe the reason for the worse performance of top-2 self-MOE comes from the excessive redundant information introduced by an irrelavant expert. But this guess contradicts your claim in line 88 that MiXSE explores synergies among experts. I wish to have a more detailed discussion about this part. Given the hypothesis that synergies among experts help to boost performance, why top-1 self-MOE outperforms top-2 self-MOE? This is important for this work, because by only choosing one expert, it largely resembles self-specialization.
2. Line 348. I am also suspicious about the claim that jointly training router and experts would make semantic distinctions among experts diluted. Have you tried to jointly train them and activate more experts (>= 2)? Could you provide it in your rebuttal (sorry I know it requires lots of computational resources)? I just think the experiments and the obervations are not fully supported by your claim. This is also related to your claim that synergies among experts happen when you have multiple experts controlled by a router in self-MOE. If you don’t jointly train them, how synergies happen?
3. Again in Section 4.3, the routing distribution shows that datasets like MMLU benefit from both knowledge and reasoning experts. So why don’t you just activate more experts?
4. Section 4.7. Have you tried adding experts in different orders? For example, R + M + C + K.
5. In Table 3, you labeled TruthfulQA as safety. I am not sure why.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730606525086

thread_type:
official_review

<thread_object>
creator: reviewer_1lag

<content>
summary:
Self-MoE constructs the corresponding experts modules to realize the self-specialization mechanism to improve the performances on non-specialized tasks and diverse benchmarks such as knowledge, reasoning, math, and coding.

strengths:
1. Introducing the computation overhead realizes better performance improvement
2. Using the self-optimized routing activates the distinct domain-specific capabilities to help improve the performance shared base LLM 
3. The presentation is good and the experiments are detailed

weaknesses:
1. The comparison with related works such as  [1] is not enough. The concept of lora-moe has been introduced in this work.  This slightly affects the novelty of the work.

[1] Zadouri T, Üstün A, Ahmadian A, et al. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning[J]. arXiv preprint arXiv:2309.05444, 2023.

questions:
1. How does the number of experts impact the downstream performance? If the knowledge for differenct experts is similar, is it possible to reduce the number of experts? 
2. How does the model generate to the unseen tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730768314081

thread_type:
official_review

<thread_object>
creator: reviewer_tfml

<content>
summary:
This paper presents a novel approach to modularize large language models by constructing expert modules from self-generated synthetic data and creating a compositional system. Unlike previous Mixture of Experts approaches that use LoRA and rely on either human-labeled data or pre-trained modules, this method develops modules from scratch, adapting the model to specific tasks. Experimental results demonstrate the advantages of this approach over base LLMs and self-specialized LLMs in multiple tasks.

strengths:
- Mitigation of Forgetting: Compared to monolithic models, which often face challenges with knowledge retention, the proposed approach maintains the integrity of each expert module, which enhances the overall model's performance and adaptability.

- Lightweight and Synthetic Data-Driven Modules: The method constructs individual, lightweight expert modules from synthetic data, which bypasses the need for human-labeled data and broadens the scope of applicability.

- Generalization: The generalization tests indicate that the Self-MoE approach offers benefits beyond the targeted tasks, improving performance on benchmarks that were not explicitly used in training.

weaknesses:
- Limited Model Sizes: The experiments primarily focus on small-scale LLMs (7B/13B models), leaving open the question of whether these findings can extend to larger models. Testing on a wider range of model sizes would strengthen the claims.
- Quality and Diversity of Synthetic Data: How does the approach ensure the correctness, diversity, and quality of the instruction-response data generated for training the expert modules?
- Domain Granularity: How is the domain granularity determined? For instance, broad domains like reasoning, math, and coding can be subdivided further (e.g., reasoning into medical, finance). To what extent do the findings depend on the chosen level of domain specificity?

questions:
- Router Details: The router is crucial to the Self-MoE model’s performance but is described only at a high level in the paper. More detailed information (e.g., training etc.) in the main body would enhance the clarity and accessibility of the approach.

- Synergies Across Domains: One reason for the Self-MoE’s superior performance might be that LLMs can exploit synergies across areas of expertise. Would this advantage persist if the domains were highly distinct, such as medical vs. finance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732312845810

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Reviewers

comment:
We thank all the reviewers for taking their valuable time to provide insightful comments and suggestions to improve our paper.

We are pleased to note that all reviewers recognized the value of our work, as evidenced by the following positive comments:
- “It's a good idea to stimulate diverse domain-specific competencies inherent within the base model …” (fc1R)
- “Novel idea of self-MOE and it can be a promising new paradigm for stronger foundation models.” (eqDV)
- “This approach offers the advantage of eliminating the necessity for domain-specific data acquisition.” (fc1R)
- “Introducing the computation overhead realizes better performance improvement” (1Lag)
- “Using the self-optimized routing activates the distinct domain-specific capabilities to help improve the performance shared base LLM” (1Lag)
- “Mitigation of Forgetting”, “Lightweight and Synthetic Data-Driven Modules”, “Generalization” (tfML) 
- “I actually like this paper.” (eqDV)
- “The writing and presentation are clear and easy to read.” (fc1R)
- “Paper is well-written.” (eqDV)
- “The presentation is good and the experiments are detailed” (1Lag)

We believe the thoughtful reviews and the recommendations made by the reviewers have substantially improved the quality of the paper. Based on the reviewers' suggestions, we have updated the paper with the following major changes:
- Added a new experiment on joint training with top-2 activation (Table 2 and Section 4.2) (eqDV)
- Included a new ablation result on layer-specific routers, comparing with a shared router (Table 2 and Section 4.2) (fc1R)
- Incorporated Instance Merging results for different base models (Figure 4 and Section 4.5) (fc1R)
- Added new analyses of self-generated data (Table 5 and Section 4.9) (fc1R)
- Reorganized Figure 1 and deleted Table 5 (full content in Table 7) to secure space for additional results (eqDV)
- Elaborated on the router in Section 3.2 (tfML)
- Elaborated on the baseline comparisons in Section 4 (fc1R)
- Provided detailed results of Self-MoE applied to different base LLMs (Table 8 in Appendix) (eqDV)
- Included visualized examples of the token-level routing decisions (Table 10 in the Appendix) (eqDV)

We hope that these changes adequately address your comments and improve the clarity and the quality of the paper. Thank you again for your thoughtful feedback and continued consideration of our work!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel approach to transform monolithic LLMs into a mixture-of-experts system with self-specialized experts using synthetic data. The key scientific claim is that Self-MoE can improve model performance across diverse domains while mitigating catastrophic forgetting, without requiring domain-specific human-labeled data. The main strengths are: (1) the novel combination of self-specialization with MoE architecture, (2) effective use of synthetic data for expert training, and (3) comprehensive empirical validation across multiple base models and tasks. The primary weaknesses include limited experimentation with larger models and some initial ambiguity around the router training mechanism. I recommend acceptance due to its novel technical contribution in combining self-specialization with MoE, strong empirical results showing consistent improvements across different base models, and the practical advantage of not requiring domain-specific labeled data.