Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts

## Abstract

Abstract We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts, named MiXSE (MiXture of Self-specialized Experts). Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data, each equipping a shared base LLM with distinct domain-specific capabilities, activated via self-optimized routing. This allows for dynamic and capability-specific handling of various target tasks, enhancing overall capabilities, without extensive human-labeled data and added parameters.
Our empirical results reveal that specializing LLMs may exhibit potential trade-offs in performances on non-specialized tasks. On the other hand, our Self-MoE demonstrates substantial improvements (6.5%p on average) over the base LLM across diverse benchmarks such as knowledge, reasoning, math, and coding. It also consistently outperforms other methods, including instance merging and weight merging, while offering better flexibility and interpretability by design with semantic experts and routing. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.

## 1 Introduction

The remarkable success of Large Language Models (LLMs) has been largely attributed to their generalist nature, allowing them to perform a wide variety of tasks (Brown et al., ; Touvron et al., ; Jiang et al., ; Team et al., ). Predominantly designed as monolithic architectures, these models rely extensively on large-scale data to embed generalized language capabilities across vast parameter spaces. While effective, this monolithic architecture, as illustrated in Figure 1, inherently suffers from significant drawbacks such as inefficiency in scaling (Zhang et al., ; Wan et al., ), susceptibility to forgetting previously learned information when adapted to specialized tasks (Kotha et al., ; Huang et al., ), and a lack of transparency which leads to the black-box nature (Zhao et al., ).

Figure: Figure 1: Concept of Self-MoE, illustrating the transformation from a monolithic LLM to a compositional system, MiXSE, without extensive resources and addition of significant parameters. MiXSE distinguishes itself from traditional MoEs and other models in post-training, lightweight semantic experts, and/or self-generated synthetic data. The results showcase MiXSE’s improved capabilities over the base LLM (e.g., Gemma-7B) across all domains, unlike the knowledge-specialized LLM that compromises other capabilities.
Refer to caption: x1.png

Meanwhile, the increasing demand to handle domain-specific or expert-level tasks has highlighted the need for specialization of LLMs (Cheng et al., ; Ling et al., ; Feng et al., ). However, effective tuning often relies on high-quality, human-annotated data, which is costly and challenging to scale (Kang et al., ), especially in specialized domains where expertise is scarce and valuable (Wu et al., ). Self-specialization (Kang et al., ) offers a promising alternative, aligning models with self-generated synthetic data. While this technique has proven effective in cross-task generalization within a target expert domain, we posit that it may compromise performance in areas outside the target domain.

In this paper, we explore the following question: How can we build compositional LLMs that enjoy versatile expertise, while using minimal resources?
We introduce Self-MoE (Figure 1), an approach that transforms a monolithic model into a compositional (Zaharia et al., ) system, called MiXSE (MiXture of Self-specialized Experts). This approach differs from prior MoE work using LoRA (Hu et al., ), which either relies on human-labeled data (Wu et al., ) or assumes the existence of trained modules (Huang et al., ; Muqeeth et al., ). Instead, our Self-MoE constructs individual lightweight expert modules from scratch using synthetic data, inspired by the concept of self-specialization. Each module is integrated with a shared base LLM, and the entire system is enhanced by a self-optimized routing mechanism. In contrast to monolithic models, which often suffer from forgetting issues when adapted or merged under fixed, static parameters, our modular design preserves the integrity and semantics of each expert. This allows for dynamic, precise handling of various target domain tasks, boosting the model’s overall capability, adaptability, and interpretability.

Through extensive empirical studies conducted across a variety of popular domains, including knowledge, reasoning, math, and coding, we find that specialization often comes with trade-offs, typically degrading performance in non-targeted domains. However, our Self-MoE demonstrates substantial overall improvements over a base LLM across all target domains without compromising performance on other tasks. Notably, the compositional nature of our MiXSE appears to exploit synergies among experts, even outperforming all individual specialized experts.

Moreover, MiXSE clearly surpasses other strong baselines such as instance merging and weight merging, under similar settings, while offering better flexibility and interpretability. Detailed analyses highlight the critical role of the routing mechanism and the contribution of semantic experts in achieving these results. Our interpretable visualizations of routing distributions further elucidate how tasks are dynamically allocated to the most relevant experts. Lastly, we further validate that there are no issues related to forgetting unlike monolithic baselines, and that our approach can be applied to various model families and sizes.
In summary, our key contributions are as follows:

- •
We highlight the inherent limitations of monolithic model specialization, where focusing on a specific capability often comes at the cost of degrading performance in other domains.
- •
We propose Self-MoE, which allows a base, monolithic LLM to upgrade into a modular system of lightweight, self-specialized experts, without requiring extensive human supervision, compute resources, or overhead in active parameters.
- •
We provide comprehensive experiments and analyses across a range of benchmarks, where Self-MoE demonstrates consistent improvements with an average of 6.5%p across domains over a base LLM, outperforming various baselines.
Our ablation studies validate the impact of modularity, routing strategies, and the use of self-generated synthetic data.
Moreover, our analyses explore routing distributions, forgetting issues, and the applicability of our approach to five different base LLMs.

Figure: Figure 2: Overview of the Self-MoE approach to building a compound system of specialized experts and a router in a self-improving manner. In the Self-Specialization phase (left side), the base LLM is aligned with self-generated synthetic data for each target specialization, producing lightweight expert modules. The right side shows MiXSE where each self-specialized expert is dynamically engaged based on the decisions of the self-optimized router.
Refer to caption: x2.png

## 2 Problem Statement

The primary focus of this work is on self-improving LLMs’ target capabilities on the fly, specifically under settings constrained by minimal resources and without the addition of significant parameters. Traditional LLMs, which are generally monolithic, require expensive human-labeled data to be better specialized, thereby limiting their adaptability and scalability when resources are constrained. We hypothesize that a modular, compositional model utilizing self-generated synthetic data for self-improvement can dramatically improve specific target capability, adaptability, and interpretability while reducing dependency on expensive human-annotated datasets.

Specifically, given a base LLM $\Theta_{0}$ and a minimal set of seed data (e.g., 100) for each of the target capabilities $\{T_{i}\}_{i=1}^{n}$ (e.g., knowledge, math), our goal is to transform $\Theta_{0}$ into an enhanced compositional model $\Theta_{comp}$ where $n$ target expert modules $\{\Delta\Theta_{i}\}_{i=1}^{n}$ are effectively integrated. Formally, the Self-MoE transformation function is defined as:

$$ $f_{trans}:(\Theta_{0},\ \{T_{i}\}_{i=1}^{n})\rightarrow\Theta_{comp}=\Theta_{0 }\ \cup\ \{\Delta\Theta_{i}\}_{i=1}^{n}$ $$

Here, under our problem setting, the number of parameters of $\Theta_{0}$ and $\Theta_{comp}$ should not be significantly different, necessitating that the expert modules $\Delta\Theta_{i}$ be lightweight (i.e., LoRA (Hu et al., )). The available seed data are limited but can be reasonably collected (e.g., 100). Importantly, we do not assume the availability of larger/teacher models at one’s hand; instead, we aim to develop a method that enables self-improvement and is designed to be universally applicable.

## 3 Method: Self-MoE

In this section, we describe Self-MoE, our proposed framework designed to build a compositional model in which specialized expert modules and a routing component are learned in a self-training manner to cooperate effectively. At a high level, Self-MoE decomposes the monolithic structure of a base LLM into a dynamic mixture of self-specialized units, each equipped with distinct target capabilities. This section outlines the overall pipeline and architecture of Self-MoE, illustrated in Figure 2, which details both the self-specialization of individual target expert modules and their integration to form a compositional system, MiXSE (MiXture of Self-specialized Experts).

### 3.1 Building Expert Modules through Self-Specialization

The first step of Self-MoE is creating specialized modules $\{\Delta\Theta_{i}\}_{i=1}^{n}$ for each target expertise, while adhering to the desiderata discussed in Section 2. That is, the modules should be lightweight and self-improving. We employ the concept of self-specialization (Kang et al., ) where a base LLM is aligned with self-generated synthetic data for target specialization, resulting in lightweight LoRA (Hu et al., ) experts.

#### Targeted Generation.

Self-specialization involves generating synthetic instruction-response data $D_{i}=\{({\text{{inst}}}^{(1)}_{i},{\text{{resp}}}^{(1)}_{i}),({\text{{inst}}}
^{(2)}_{i},{\text{{resp}}}^{(2)}_{i}),...\}$ tailored to each target domain $T_{i}$. We ensure the data is both diverse and highly relevant to the specialized tasks/domains each module will address. The generation includes the following steps:

(1) Seed Construction: First, given a target $T_{i}$ identified, we prepare a small number of seed examples (e.g., 100) that capture essential characteristics and scenarios relevant to each target domain $T_{i}$. While we exploit existing datasets for the purpose of demonstration, we posit manual annotation for such a small number should be reasonable in real-world applications. These seeds serve as the foundational dataset from which synthetic variations are generated.

(2) Instruction Brainstorming: Once the seed examples are established, the next step is to diversify the range of instructions (and corresponding input contexts) through a brainstorming process. Specifically, we prompt(^1^11The prompts can be found in Table 9-12 in Appendix.) a base LLM $\Theta_{0}$ to create new instructions following sequences of seed instructions given in-context.

(3) Response Generation: The final step involves generating corresponding responses for the newly created instructions. We use seed instruction-response pairs as in-context demonstrations to extract latent relevant knowledge from $\Theta_{0}$.

#### Self-Align with LoRA

With each specialized synthetic data $D_{i}$ in place, we now proceed with the self-alignment of $\Theta_{0}$ to induce specialization, separately producing lightweight expert components $\Delta\Theta_{i}$. Note that $D_{i}$ are self-generated by $\Theta_{0}$ and used to specialize the same $\Theta_{0}$ using an adapter module $\Delta\Theta_{i}$, resulting in an specialized model $\Theta_{spec}=\Theta_{0}+\Delta\Theta_{i}$.
Specifically, we utilize Low-Rank Adaptation (LoRA) (Hu et al., ), which integrates additional trainable parameters that are specific to each domain $T_{i}$ while keeping $\Theta_{0}$ intact.
Within the corresponding $\Theta$, we define $\theta$ as the weights at a certain layer where LoRA is attached.
Let $\theta_{spec}\in\mathbb{R}^{d\times k}$ be updated weights at a specific LoRA layer which can be decomposed as:

$$ $\displaystyle\theta_{spec}$ $\displaystyle=\theta_{0}+\Delta\theta_{i}$ $\displaystyle=\theta_{0}+\theta_{B_{i}}\theta_{A_{i}}$ $$

where $\theta_{B_{i}}\in\mathbb{R}^{d\times rank}$ and $\theta_{A_{i}}\in\mathbb{R}^{rank\times k}$, with $rank\ll\min(d,k)$.
The forward pass becomes:

$$ $h=\theta_{spec}x=\theta_{0}x+\theta_{B_{i}}\theta_{A_{i}}x$ $$

This applies to all LoRA layers, and only $\Delta\Theta_{i}=\{\Delta\theta_{i}^{(1)},\Delta\theta_{i}^{(2)},...\}$ is updated during training using $D_{i}$.
As a whole, this process of self-specialization can be defined as producing an expert module $\Delta\Theta_{i}$ for the $i$-th target along with the corresponding synthetic data $D_{i}$ (Left in Figure 2):

$$ $f_{ss}:(\Theta_{0},T_{i})\rightarrow(\Delta\Theta_{i},D_{i})$ $$

We iterate this process for each target domain, focusing on knowledge, reasoning, math, and coding.

### 3.2 Mixture of Self-Specialized Experts

After each expert module is individually specialized through the self-specialization process, they are integrated into a compound system $\Theta_{comp}$, MiXSE (MiXture of Self-specialized Experts). MiXSE is designed to leverage the distinct capabilities of each module, orchestrating their cooperation to handle diverse tasks dynamically and efficiently. To achieve this benefit, a router module $\theta_{r}$ is also incorporated, which analyzes each input token to dynamically route to the most appropriate expert module based on the task at hand.

Specifically, within each layer, the output $h$ for each input $x$ is calculated by combining the contributions of the selected expert modules $\Delta\theta_{i}$, weighted by their relevance as determined by the router:

$$ $\displaystyle h$ $\displaystyle=\theta_{0}x+\sum\nolimits_{i=1}^{n}\alpha_{i}\Delta\theta_{i}x$ $\displaystyle=\theta_{0}x+\sum\nolimits_{i=1}^{n}\alpha_{i}\Delta\theta_{B_{i} }\theta_{A_{i}}x$ $$

where $\alpha$ represents a set of weights computed by the router (i.e., a linear layer) $\theta_{r}\in\mathbb{R}^{n\times k}$.

$$ $\alpha=\text{top-k(softmax}(\theta_{r}x))$ $$

Note that we only take top-k probabilities and mask out the others to efficiently reduce computation. In essence, this also allows the pre-trained base weights $\theta_{0}$ to be sufficiently able to contribute, mitigating potential issues of over-specialization such as forgetting or diminished generalizability.
The router $\theta_{r}$ is a linear layer, shared across all LoRA layers, and is trained using the aggregated self-generated data $D=\{D_{i}\}_{i=1}^{n}$ to learn how to optimally select modules for a given task:

$$ $\displaystyle L(\theta_{r})=-\mathbb{E}_{({\text{{inst}}},\ {\text{{resp}}}) \sim D}[logP_{\Theta_{0}}({\text{{resp}}}\ |\ {\text{{inst}}};\ \theta_{r},\{ \Delta\Theta_{i}\}_{i\text{=1}}^{n})]$ $$

Here, we solely optimize the router to preserve the explicit semantic distinction of expert modules.

## 4 Experiments and Results

#### Datasets.

We evaluate Self-MoE across diverse domains categorized into knowledge, reasoning, math, and coding: MMLU (0- & 5-shot) (Hendrycks et al., ), BBH (3-shot) (Suzgun et al., ), GSM8K (8-shot) (Cobbe et al., ), and HumanEval (0-shot) (Chen et al., ), respectively. For MMLU, we primarily employ the 0-shot setting unless otherwise specified, based on established observations (Dettmers et al., ; Lin et al., ) that tuning yields only marginal effects in the 5-shot setting for this task.
To test generalization (Section 4.4), we additionally evaluate on MATH (4-shot) (Hendrycks et al., ), MBPP (3-shot) (Austin et al., ), NaturalQuestions (5-shot) (Kwiatkowski et al., ), TriviaQA (5-shot) (Joshi et al., ), Hellaswag (0-shot) (Zellers et al., ), PIQA (0-shot) (Bisk et al., ), and TruthfulQA (0-shot) (Lin et al., ).

#### Baselines.

To assess the effectiveness of Self-MoE, we compare performance against several baselines that use the same number of active parameters during inference:

- •
Four Self-Specialized Models (Kang et al., ): Trained on self-generated synthetic data for individual domains: knowledge, reasoning, math, and coding.
- •
Instance Merging (Multi-Task Tuning) (Chung et al., ): Leverages the aggregated synthetic data generated by self-specialization to train a model capable of handling multiple tasks.
- •
TIES (Yadav et al., ), DARE (Yu et al., ): Advanced weight merging methods integrating multiple expert strengths into a unified model.

We also contextualize these results with computationally intensive methods reported in the literature, despite indirect comparisons: BTM (Li et al., ), Sparse Upcycling (Komatsuzaki et al., ), BTX (Sukhbaatar et al., ), GLAN (Li et al., ), Orca (Mitra et al., ), and Merlinite (Sudalairaj et al., ) in Appendix D.1.

#### Implementation Details.

We adopt Gemma-7B (Team et al., ) as a base LLM for our main experiments, and additionally apply Self-MoE to various models, such as LLaMA-2 7B & 13B (Touvron et al., ), Mistral 7B (Jiang et al., ), and LLaMA-3 8B (AI@Meta, ) in Section 4.5. We use 100 seeds to generate 5K synthetic data for each domain, resulting in 20K data. Each LoRA module contributes less than 0.3% to the parameters of the base model, and the router’s parameters are negligible, resulting in the added parameters of MiXSE amounting to only about 1%.

Table: Table 1: Main results. All models are built upon the same base LLM, Gemma-7B, taking self-improving approaches and having the same active parameters during inference. Corresponding aligned performances of self-specialization are underscored. Each column’s best performance is highlighted in bold, while the gains achieved by our MiXSE over the base LLM are indicated.

### 4.1 Main Results

In Table 1, we showcase comparative benchmark results of various approaches across four specialized domains: knowledge, reasoning, math, and coding. All baselines use self-generated synthetic data based on the same Base LLM, Gemma-7B, and LoRA for tuning to ensure fair comparisons.

First, we confirm self-specialization markedly enhances target-specific expertise, compared to the base LLM. For instance, we can see substantial gains from corresponding specialized models (e.g., Knowledge Self-Spec. in the knowledge domain): 58.4 to 64.0 in knowledge, 56.1 to 60.2 in reasoning, and so on. However, this focused improvement sometimes comes at the cost of reduced performance in non-targeted areas, as evidenced by the drop in scores for the Knowledge Self-Spec. model in reasoning, math, and coding. This trade-off highlights the inherent limitation of over-specialization. In contrast, our MiXSE, demonstrates consistent improvements across all domains, due to its modular, compositional architecture that makes use of dynamic routing to leverage optimal experts. Surprisingly, it even outperforms all corresponding specialized models, indicating that it effectively synergizes the strengths of each specialization.

In comparison with other static merging methods like Instance Merging, TIES, and DARE, MiXSE stands out for its superior adaptability. While they attempt to combine the strengths of different specialization areas into a single model, they lack the dynamic flexibility that MiXSE offers. Notably, simple instance merging (i.e., multi-task tuning), though effective in enhancing the base LLM across domains, still falls short of achieving the superior average performance of 54.3 seen with MiXSE. This validates the advantages of dynamic expert integration in a compositional system.

### 4.2 Ablation Study

Now that we have verified the effectiveness of MiXSE as a whole, we evaluate the impact of different configurations and components of the system, presented in Table 2. The configurations vary in terms of routing strategies and integration of experts, offering insights into the contributions of each element to the system’s overall effectiveness.

We start by examining the Top-k routing strategy, which plays a crucial role in our model. Our findings show that both the Top-1 and Top-2 expert configurations deliver the best performance. This suggests that identifying and leveraging the most relevant expert for a given task is typically sufficient and most effective. On a side note, the similar performances of the different configurations may highlight the robustness of our method. Given the similar performances, we prefer the Top-1 expert setup for better efficiency.

Interestingly, the results also indicate a drop in performance when using All Experts. This can be attributed to that involving all experts regardless of their relevance can introduce noise and dilute the specific contributions of the most pertinent experts. Additionally, involving more experts than necessary can increase computational overhead.

Furthermore, employing Random Routing serves as a useful setup to highlight the effectiveness of strategic expert selection of our Self-Optimized Router, which is a key component of our MiXSE. We observe that the performance significantly decreases under this configuration, highlighting the router’s role in dynamically tailoring the selection of experts according to the specific requirements of each task. The router’s ability to discern and activate the most suitable experts based on the context is critical for optimizing performance. Notably, this ability is learned by relying on a very small amount of seed data.

Another interesting finding comes from the configuration where experts and the router are jointly trained, which means that the semantic distinctions among experts may be diluted.
This setup substantially decreases performance relative to scenarios where the router and experts are optimized independently. This decline validates that semantic experts play a crucial role in enhancing the system’s capability to handle tasks requiring specific expertise, while offering better interpretability (Section 4.3).

Table: Table 2: Analysis and ablation of the router in our MiXSE. Configurations vary to investigate the optimal number of experts used, to verify the possibility of self-learning for the router, and to see the importance of semantic distinctions among experts within the compositional system.

Figure: Figure 3: Routing analysis that shows routing distributions over four domains for each benchmark, averaging the weights across tokens within individual tasks.
Refer to caption: x3.png

### 4.3 Routing Analysis

Understanding how MiXSE allocates tasks to its various experts is crucial for gauging its interpretability. By analyzing the routing distributions across four distinct domains, we aim to see whether the system matches queries to the most suitable experts. Figure 3 presents the routing distributions used to solve each benchmark, where the weights are averaged across tokens and layers within individual tasks.

We first observe that the MiXSE’s router effectively selects the correct expert for each corresponding target. This is evident from the impressive alignment between tasks and the experts chosen by the router; for example, the knowledge expert predominantly handles knowledge tasks, while the coding expert is routed coding tasks. This highlights the router’s ability to learn and apply this routing automatically and consistently, making the system’s decisions interpretable and trustworthy.

Beyond the direct matching of tasks to domain-specific experts, the router also demonstrates its ability to exploit synergies between different areas of expertise. For instance, the reasoning expert is frequently involved in tasks across the knowledge, math, and coding, reflecting the system’s compositional use of expertise. This explains the reason for MiXSE’s superior performances across all domains even beyond all specialized modules in Table 1.

Table: Table 3: Investigation on generalization and a forgetting issue of Self-MoE. Non-Target (In-Expertise) indicates where MiXSE does not directly specialize using seed data directly while relevant to targets. Non-Target (Out-of-Expertise) refers to irrelevant cases.

### 4.4 Generalizability Test

While Self-MoE has shown clear benefits in target benchmarks such as MMLU, BBH, GSM8K, and HumanEval, one may be curious about its generalizability to non-targets, or concerned with the potential issues of specialization such as forgetting. In Table 3, we conduct an investigation using non-targeted benchmarks that were not utilized in building MiXSE.

On MATH and MBPP benchmarks, which can be considered highly relevant to target benchmarks, GSM8K and HumanEval, we find our Self-MoE can still improve over the base LLM even though they were not directly targeted in our training regime. This finding supports the generalizability of the Self-MoE approach.

Concerning the potential side effect of forgetting, we extend our testing to include domains such as world knowledge, common sense, and safety, which are rarely associated with the targets directly.
Our experiments show that overall, there are rarely meaningful performance drops when applying our Self-MoE. Only a minor drop is observed with MiXSE in TriviaQA, but this is substantially less than in the case of instance merging.
This suggests our approach almost maintains existing knowledge for non-targets while significantly boosting target performances, unlike monolithic baselines.

Figure: Figure 4: Results of Self-MoE w/ other LLMs.
Refer to caption: x4.png

### 4.5 Applicability to Other Base LLMs

Following the successful demonstration of our Self-MoE approach based on Gemma-7B, we now present Figure 4 where we apply Self-MoE to other base LLMs beyond Gemma-7B. We use diverse model variants including LLaMA-2 7B & 13B, Mistral 7B, and LLaMA-3 8B. Our findings suggest that our approach improves all models regardless of the model family, size, and level of base performance. This is significant as it might imply that one can take any monolithic model to enjoy a free upgrade to a compositional system that offers better effectiveness, flexibility, and interpretability.

Figure: Figure 5: Analysis with the varied sizes of self-generated synthetic data for Self-MoE.
Refer to caption: x5.png

### 4.6 Impact of the Number of Synthetic Data

Figure 5 illustrates the impact of scaling self-generated synthetic data for Self-MoE. As the data scales from 0 to 20K, our MiXSE model demonstrates substantial and consistent improvements over the base one in average performance across domains, suggesting the scalable potential of Self-MoE.
Instance Merging, serving as a strong baseline, also benefits from increased data, but the gains progress at a slower rate, as evidenced by linear trendlines. This reflects the inefficiency of the static merging scheme, which, being monolithic, suffers from trade-offs in knowledge gains and forgetting.

Table: Table 4: Scaling the number of experts. K: Knowledge expert. R: Reasoning expert. M: Math expert. C: Coding expert.

### 4.7 Scaling the Number of Experts

In Table 4, we present the results of MiXSE composed of varying numbers of experts, with experts added progressively one at a time in the order of knowledge, reasoning, math, and coding.
The results indicate that starting with the knowledge expert, which initially exhibits a performance trade-off, subsequent additions of reasoning, math, and coding experts consistently enhance overall performance. This highlights the compositional MiXSE’s advantage of adaptability and modularity.

### 4.8 Discussion on the Overhead of Self-MoE

One possible concern in adapting LLMs into compositional systems using Self-MoE is the potential introduction of overhead. Here, we discuss this aspect in detail, emphasizing that the additional overhead of Self-MoE is minimal while yielding significant performance gains. Essentially, the expert modules in Self-MoE are lightweight LoRA modules, contributing only about 1% additional parameters (total) for four experts, as detailed in Table 5 (Total Params). These experts are sparsely activated, which maintains low active parameters (7B + 0.3%) during inference, thus efficiently minimizing inference overhead. In contrast, traditional MoE models like Mixtral (Jiang et al., ) and BTX (Sukhbaatar et al., ) typically employ a feedforward network (FFN) layer for each expert, resulting in a significant proportional increase in total parameters as the number of experts grows, as indicated in Table 5, which demands much more memory for model loading. The design choice in Self-MoE leads to better scalability and resource efficiency, especially when the number of experts is scaled to incorporate numerous domains of expertise.

## 5 Related Work

To offer a broader perspective, Table 5 presents a comprehensive summary of various models that, while relevant, are not directly comparable. For further discussions and a more detailed comparison, please refer to Appendix D.1.

#### Combination of Experts.

There have been numerous efforts to combine the strengths of multiple models or modules. The Mixture of Experts (MoE) models such as Switch Transformer (Fedus et al., ), GLAM (Du et al., ), and Mixtral (Jiang et al., ) exemplify this, dynamically allocating tasks based on the expertise of each component for better efficiency and scalability. These models contrast with ours by not prioritizing lightweight experts, resulting in a larger model with more parameters. Unlike their experts implicitly learned during pre-training, Self-MoE explicitly creates semantic experts for targeted improvements.

Another relevant area is merging, involving the weighted averaging of multiple models to form a single, aggregated model (Wortsman et al., ; Matena & Raffel, ; Ilharco et al., ; Jin et al., ). One of the leading methods, TIES (Yadav et al., ) tackles conflicts and parameter inconsistencies among models. DARE (Yu et al., ) further reduces the redundancy of parameters. However, these methods are fundamentally static in that they operate with fixed parameters once merged, which may lead to interference, lacking the dynamic flexibility that MiXSE offers.

There exist notable recent MoE models that similarly explore the utilization of semantic experts, albeit in distinct contexts (Gururangan et al., ; Wu et al., ; Muqeeth et al., ; Sukhbaatar et al., ).
MOLE relies on human-labeled data, and PHATGOOSE assumes the availability of existing expert models trained by external creators and necessitates additional training for a router on the creators’ side. DEMix and BTX rely on extensive pre-training, demanding significant resources, yet it as a pre-trained model holds the potential to complement our self-training approach. Unlike MOLE and PHATGOOSE, our Self-MoE framework creates experts and a router from scratch through self-improvement, while using minimal resources, as contrasted to DEMix and BTX.

Table: Table 5: Comprehensive summary of relevant models for references. Detailed discussions are provided in Appendix D.1.

#### Self-Improvement and Specialization of LLMs.

The pursuit of enhancing the capabilities of LLMs often revolves around an instruction-tuning scheme, which can significantly boost cross-task generalizability (Ouyang et al., ; Su et al., ; Mishra et al., ; Wei et al., ). Due to the bottlenecks of expensive annotation costs which lead to limited scalability, the self-training concept (Luo, ) has gained attention from the community, where LLMs are aligned with automatically self-generated synthetic instructions (Wang et al., ; Sun et al., ; Li et al., ). These are distinguished from distillation techniques (Hinton et al., ; Kang et al., ), which assume a stronger teacher model (Mitra et al., ; Li et al., ; Sudalairaj et al., ), limiting their applicability.

With the growing need to adapt generalist models to specific domains, Kang et al. () adopts the self-training for specialization, tackling that general instruction tuning is rarely effective in expert domains. While this work lays a foundation for enhancing specialized expertise with minimal resources, we recognize inherent trade-offs in a monolithic structure, such as performance compromises outside specialized domains.
Conversely, our Self-MoE achieves uncompromising multiple expertise with a modular approach without extensive resources and adding many parameters.

## 6 Conclusion

In this study, we proposed Self-MoE to build compositional LLMs with self-specialized experts, MiXSE, to enhance targeted capabilities, adaptability, and interpretability without the reliance on extensive human-labeled data. Empirical evaluations across diverse domains with multiple base models demonstrated that MiXSE significantly enhances base LLM performance and overcomes specialization trade-offs. We believe this work offers a step towards modular, self-improving paradigms which can address the inherent limitations of monolithic models, providing a promising direction for future LLM research.

## Appendix A Experiment Details

We provide each of our self-specialization prompts for knowledge, reasoning, math, and coding experts in Tables 9, 10, 11, and 12. We largely follow Kang et al. ()’s prompt structure to ensure quality, with additional domain-specific instructions that inform task-related information.

For our evaluation, we employ popular and widely accepted evaluation frameworks to pursue standard evaluation setups and protocols: HELM (Liang et al., ), LM Evaluation Harness (Gao et al., ), and BigCode Evaluation Harness (Ben Allal et al., ). We use Huggingface PEFT (Mangrulkar et al., ) and XLoRA (Buehler & Buehler, ) for the implementation of MoE compatible with LoRA.

Regarding seed instructions, we sampled 100 training instances from each of the MMLU, BBH, and GSM8K datasets, for knowledge, reasoning, and math domains, respectively. For coding, since the size of the HumanEval dataset is very small and thus the training set is not available, we took 100 samples from the MBPP training set and converted the task format to make them suit the HumanEval.

During instruction generation, we use three seed data, which are randomly sampled, as in-context examples, using a temperature of 1 and top-p of 0.98, whereas we use five seed data in-context for response generation with greedy decoding.
For specialization, we use LoRA applied to all modules with a rank of 8 and alpha of 16, and train it using a learning rate of 3e-4, epochs of 3, and batch size of 32. We train each module and MiXSE using a standard Alpaca (Taori et al., ) prompt template on a single A100-80GB, which takes only a few hours.

## Appendix B Limitations

While our study demonstrates promising results for the Self-MoE, we recognize areas requiring further investigation in future work. Employing self-specialization Kang et al. () to generate synthetic data within our framework may raise concerns about potential data contamination and noise. Nonetheless, findings from Kang et al. (), which conducted an n-gram overlap analysis between the self-specialization data and test data, confirmed no significant overlap, thus alleviating the concerns about contamination. Despite this, the need for continuous monitoring of potential biases from pre-training and the development of enhanced data validation and noise filtering strategies remain important, and may present interesting direction for future work. Moreover, due to computational constraints, we did not scale our model and data to their full potential. We also did not work on the optimization of the XLoRA, the MoE module we used, to focus purely on the research problem defined in this study. Future work should therefore concentrate on overcoming these limitations, which will enable better data quality and more extensive training to unveil the full potential of the Self-MoE framework.

Table: Table 6: Dataset statistics. Non-Target (In-Expertise) indicates where MiXSE does not directly specialize using seed data directly while relevant to targets. Non-Target (Out-of-Expertise) refers to irrelevant cases.

Table: Table 7: Additional comparisons with other models for references. Results are extracted from each corresponding paper, except for pre-training methods where the numbers are all from BTX (Sukhbaatar et al., ).

## Appendix C Dataset Descriptions

The statistics for each dataset are provided in Table 6.
The target datasets used are as follows:

- •
MMLU (Massive Multitask Language Understanding) (Hendrycks et al., ): A collection of 57 academic knowledge tasks.
- •
BBH (BIG-Bench Hard (Suzgun et al., ): A set of 27 challenging reasoning tasks.
- •
GSM8K (Grade School Math 8K) (Cobbe et al., ): A diverse set of grade school math word problems.
- •
HumanEval (Chen et al., ): A hand-written evaluation set for python programming problems.

## Appendix D Additional Results

### D.1 Additional Comparison and Discussion

In Table 7, we present additional comparisons with various other models and methods to provide a broader perspective, though comparisons may not appear to be direct, due to factors involved such as parameters, resources, etc. We discuss some noteworthy points.

Notably, although MiXSE significantly improves upon its base model, Gemma 7B, it does not yet reach the performance levels of the more powerful Mixtral 8x7B. It is important to understand that Mixtral also utilizes an MoE (Mixture of Experts) architecture, but unlike MiXSE, it does not prioritize lightweight experts, leading to a much larger model with significantly more parameters. Moreover, while Mixtral’s experts are implicitly built during pre-training, MiXSE explicitly creates semantic experts, allowing for targeted improvements and clearer interpretability. Importantly, our self-improving method can be potentially applied on top of any pre-trained model including Mixtral in principle.

Similarly, BTX (Branch-Train-MiX) uses a pre-training MoE strategy where parameter-heavy semantic experts are employed, yielding substantial enhancements over the base LLM. This approach highlights the effectiveness of using semantically rich experts to refine the model’s capabilities. To make comparisons in terms of efficiency, our model uses fewer parameters (7B), compared to BTX (12B active with much more whole parameters) and requires only about 1 GPU day for training, compared to 900 GPU days for BTX. In essence, since BTX is also a pre-training method while specialized, we expect it to be complementary to our Self-MoE, as evidenced in previous work (Kang et al., ).

With a shared spirit, MOLE and PHATGOOSE build a MoE (Mixture of Experts) using LoRA, which is semantic and lightweight. However, there are significant differences in foundational assumptions: MOLE depends on human-labeled data, while PHATGOOSE requires access to pre-trained expert models developed externally. In contrast, our Self-MoE framework independently constructs both experts and a router entirely from scratch, focusing on self-improvement without such dependencies. While their scenarios are considered reasonable in a certain context, we aim for broader applicability by minimizing assumptions on conditions.

Lastly, GLAN demonstrates outstanding performance across various domains. This is attributed to their reliance on distilling from the larger and stronger model, GPT-4, using a huge amount of data (e.g., 10 million). As outlined in our problem statement (Section 2), we deliberately avoid assuming the availability of such advanced models to ensure the broader applicability of our method which self-improves from scratch. Consequently, while acknowledging each of their own value, it is crucial to recognize that direct comparisons may not be entirely appropriate, given the fundamental differences in resource assumptions and initial conditions.

Table: Table 8: Results of MiXSE using only seed data. Seed Only training shows only marginal improvements over the Base LLM in some benchmarks, validating that the effect of Self-MoE is not merely due to the use of seed data.

### D.2 MiXSE using Only Seed Data

Table 8 shows the results of the MiXSE when exploiting only seed data for training, clarifying the benefits derived from our methodological enhancements beyond the mere inclusion of seed data in training. While the Seed Only shows slight improvements over the Base LLM in some benchmarks, the significant enhancements of our MiXSE across all benchmarks confirm that the enhanced capabilities of Self-MoE are not merely due to the use of seed data. This further highlights the achievement of self-improvement with our method.

### D.3 Vaildity of Comparative Results

In an effort to address the concern related to the sensitivity of in-context learning (Min et al., ), we perform three runs with the different lists of few-shot samples where applicable. As a result, we see that the mean of the base LLM (Gemma-7B)’s average performance across domains is 47.9 with a standard deviation (SD) of 0.56, that of our MiXSE is 53.6 with an SD of 0.60, and that of instance merging is 51.6 with an SD of 0.87. A statistical analysis between MiXSE and instance merging yields a p-value of 0.03, confirming the significant difference.

Table: Table 9: Prompts for knowledge-related instruction and response generation.

Table: Table 10: Prompts for reasoning-related instruction and response generation.

Table: Table 11: Prompts for math-related instruction and response generation.

Table: Table 12: Prompts for coding-related instruction and response generation.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Router Details: The router is crucial to the Self-MoE model’s performance but is described only at a high level in the paper. More detailed information (e.g., training etc.) in the main body would enhance the clarity and accessibility of the approach.

- Synergies Across Domains: One reason for the Self-MoE’s superior performance might be that LLMs can exploit synergies across areas of expertise. Would this advantage persist if the domains were highly distinct, such as medical vs. finance?
soundness: 3
strengths: - Mitigation of Forgetting: Compared to monolithic models, which often face challenges with knowledge retention, the proposed approach maintains the integrity of each expert module, which enhances the overall model's performance and adaptability.

- Lightweight and Synthetic Data-Driven Modules: The method constructs individual, lightweight expert modules from synthetic data, which bypasses the need for human-labeled data and broadens the scope of applicability.

- Generalization: The generalization tests indicate that the Self-MoE approach offers benefits beyond the targeted tasks, improving performance on benchmarks that were not explicitly used in training.
summary: This paper presents a novel approach to modularize large language models by constructing expert modules from self-generated synthetic data and creating a compositional system. Unlike previous Mixture of Experts approaches that use LoRA and rely on either human-labeled data or pre-trained modules, this method develops modules from scratch, adapting the model to specific tasks. Experimental results demonstrate the advantages of this approach over base LLMs and self-specialized LLMs in multiple tasks.
weaknesses: - Limited Model Sizes: The experiments primarily focus on small-scale LLMs (7B/13B models), leaving open the question of whether these findings can extend to larger models. Testing on a wider range of model sizes would strengthen the claims.
- Quality and Diversity of Synthetic Data: How does the approach ensure the correctness, diversity, and quality of the instruction-response data generated for training the expert modules?
- Domain Granularity: How is the domain granularity determined? For instance, broad domains like reasoning, math, and coding can be subdivided further (e.g., reasoning into medical, finance). To what extent do the findings depend on the chosen level of domain specificity?

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How does the number of experts impact the downstream performance? If the knowledge for differenct experts is similar, is it possible to reduce the number of experts? 
2. How does the model generate to the unseen tasks?
soundness: 3
strengths: 1. Introducing the computation overhead realizes better performance improvement
2. Using the self-optimized routing activates the distinct domain-specific capabilities to help improve the performance shared base LLM 
3. The presentation is good and the experiments are detailed
summary: Self-MoE constructs the corresponding experts modules to realize the self-specialization mechanism to improve the performances on non-specialized tasks and diverse benchmarks such as knowledge, reasoning, math, and coding.
weaknesses: 1. The comparison with related works such as  [1] is not enough. The concept of lora-moe has been introduced in this work.  This slightly affects the novelty of the work.

[1] Zadouri T, Üstün A, Ahmadian A, et al. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning[J]. arXiv preprint arXiv:2309.05444, 2023.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Line 223. I am just wondering if the top-k operation is after doing softmax, then the probability of expert selection is not normalized, meaning that when k is small alpha will also be small. Do you think it will lead to the experimental results which show that top-1 self-MOE is better than top-2 self-MOE (as presented in Table 2). I think by not normalizing the selection probability, the amount of information introduced by experts will be different when different numbers of experts are activated. So directly comparing top-1 and top-2 self-MOE is a bit unfair. Maybe the reason for the worse performance of top-2 self-MOE comes from the excessive redundant information introduced by an irrelavant expert. But this guess contradicts your claim in line 88 that MiXSE explores synergies among experts. I wish to have a more detailed discussion about this part. Given the hypothesis that synergies among experts help to boost performance, why top-1 self-MOE outperforms top-2 self-MOE? This is important for this work, because by only choosing one expert, it largely resembles self-specialization.
2. Line 348. I am also suspicious about the claim that jointly training router and experts would make semantic distinctions among experts diluted. Have you tried to jointly train them and activate more experts (>= 2)? Could you provide it in your rebuttal (sorry I know it requires lots of computational resources)? I just think the experiments and the obervations are not fully supported by your claim. This is also related to your claim that synergies among experts happen when you have multiple experts controlled by a router in self-MOE. If you don’t jointly train them, how synergies happen?
3. Again in Section 4.3, the routing distribution shows that datasets like MMLU benefit from both knowledge and reasoning experts. So why don’t you just activate more experts?
4. Section 4.7. Have you tried adding experts in different orders? For example, R + M + C + K.
5. In Table 3, you labeled TruthfulQA as safety. I am not sure why.
soundness: 3
strengths: 1. Novel idea of self-MOE and it can be a promising new paradigm for stronger foundation models.
2. Paper is well-written.
summary: This paper proposes a new paradigm of self-MOE that designs a compositional LLM integrating various experts specialized in different aspects and employing a trained router to dynamically choose experts during inference. The experts are trained with LoRA and only a small proportion of parameters are adjusted during self-specialization, keeping a high efficiency. Self-MOE has been tested on several popular LLMs, e.g., Gemma and LLaMA 3, proving its strong ability to enhance current LLMs' performance when dealing with tasks in various aspects simultaneously.
weaknesses: 1. Some claims are not very intuitive. I have questions about how synergies among expert happen and given the synergies, why only 1 expert is activated. Please refer to the questions below.
2. Some parts in the main body of this paper are not necessary, for example Table 5. I would rather see it in the appendix. Some figures are either too huge or loosely-structured and leave lots of wasted blank space (e.g., Figure 1 and 2). I think you can put more experiments about self-MOE in the main body. Please refer to point 2 of the questions below. And I would also prefer to see more detailed analysis about other base LLMs. Currenlty I only see a very abbreviated figure (Figure 4) related to this. I think breaking down their performances into several parts as how you present Gemma 7B is meaningful.

I actually like this paper. Please kindly address my concerns and questions. I think a good rebuttal will make me change my mind and raise the score. Good luck with the rebuttal.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In the instruction brainstorming stage, how to generate diverse instructions within a given domain? Since the generator is not a strong instruction-following model, e.g. GPT-4, using self-instruct approach can lead to low diversity in the generated domain-specific datasets as illustrated in [5].

2. In the mixture of self-specialized experts stage, router network parameters are shared across all layers. However, when router networks in different MoE layers do not share parameters, they typically exhibit distinct routing behaviors. What is the influence on overall performance and routing behaviors if separate, layer-specific router networks are employed instead of shared ones?

3. In the main results, the baseline methods involve fine-tuning both the LoRA and router parameters. However, this approach presupposes that all base models should be implemented using the MoE architecture, potentially introducing additional complexities in router training. To ensure a more comprehensive and equitable comparison, it is essential to add additional baseline methods, which are directly fine-tuned on the base model using different datasets (specific capabilities, mixture of capabilities, model merging).

4. In Section 4.5, can the author provide the instance merging results (using all the 20k data to direct fine-tune the base model with LoRA) for different base models?

[5] Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration

I would be happy to discuss further with the authors and reassess my score based on the rebuttal stage.
soundness: 3
strengths: 1. It's a good idea to stimulate diverse domain-specific competencies inherent within the base model and reassemble these specific experts to achieve a more powerful MoE system. 
2. This approach offers the advantage of eliminating the necessity for domain-specific data acquisition.
3. The writing and presentation are clear and easy to read.
summary: This paper proposes a self-specification method, named Self-MoE, to transform a monolithic LLM to a MoE system with self-specific experts. To construct expert modules for specific domains, this paper utilizes self-generated data from the base LLM. The resulted Self-MoE demonstrates substantial improvements over the base LLM, even achieves better performance than data mixture and weight merging.
weaknesses: 1. The primary contribution of this paper is constructing a MoE system from multiple domain experts, which are fine-tuned on the domain-specific synthesis data from base model itself. However, this self-specialization method was originally proposed by [1]. Moreover, recent studies already propose the construction of MoE system based on multiple domain-specific dense models, which can be fine-tuned using LoRA [2] or full-parameter tuning [3]. 

2. A critical question in utilizing self-synthesized data for self-improvement is where the performance improvements comes from. For instance, the improvements in [4] is based on an additional reward model to differentiate between good and bad samples. However, the underlying mechanism of Self-MoE's self-improvement remains ambiguous. It is not immediately apparent why domain-specific datasets generated by the base model, without guidance from a stronger model or reward model, can lead to overall performance gains.

3. The expert modules in Self-MoE are fine-tuned from self-synthesized domain data, which determines the upper bound of the expert's capabilities. Therefore, more analysis of the self-synthesized datasets is needed, e.g., data diversity, complexity, and the model performance compared to using existing open-source datasets.

[1] Self-specialization: Uncovering latent expertise within large language models.

[2] Mixture of lora experts.

[3] Branch-train-mix: Mixing expert llms into a mixture-of-experts llm.

[4] Direct Nash optimization: Teaching language models to self-improve with general preferences.

Meta-review: This paper proposes a novel approach to transform monolithic LLMs into a mixture-of-experts system with self-specialized experts using synthetic data. The key scientific claim is that Self-MoE can improve model performance across diverse domains while mitigating catastrophic forgetting, without requiring domain-specific human-labeled data. The main strengths are: (1) the novel combination of self-specialization with MoE architecture, (2) effective use of synthetic data for expert training, and (3) comprehensive empirical validation across multiple base models and tasks. The primary weaknesses include limited experimentation with larger models and some initial ambiguity around the router training mechanism. I recommend acceptance due to its novel technical contribution in combining self-specialization with MoE, strong empirical results showing consistent improvements across different base models, and the practical advantage of not requiring domain-specific labeled data.

Decision: Accept (Poster)