Self-MoE: Towards Compositional Large Language Models with
  Self-Specialized Experts
Authors: name: Junmo Kang; name: Leonid Karlinsky; name: Hongyin Luo; name: Zhen Wang; name: Jacob Hansen; name: James Glass; name: David Cox; name: Rameswar Panda; name: Rogerio Feris; name: Alan Ritter
Published: 2024-06-17 19:06:54

Abstract: We present Self-MoE, an approach that transforms a monolithic LLM into a
compositional, modular system of self-specialized experts, named MiXSE (MiXture
of Self-specialized Experts). Our approach leverages self-specialization, which
constructs expert modules using self-generated synthetic data, each equipped
with a shared base LLM and incorporating self-optimized routing. This allows
for dynamic and capability-specific handling of various target tasks, enhancing
overall capabilities, without extensive human-labeled data and added
parameters. Our empirical results reveal that specializing LLMs may exhibit
potential trade-offs in performances on non-specialized tasks. On the other
hand, our Self-MoE demonstrates substantial improvements over the base LLM
across diverse benchmarks such as knowledge, reasoning, math, and coding. It
also consistently outperforms other methods, including instance merging and
weight merging, while offering better flexibility and interpretability by
design with semantic experts and routing. Our findings highlight the critical
role of modularity and the potential of self-improvement in achieving
efficient, scalable, and adaptable systems.