Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
Authors: name: Haoyuan Wu, name: Haoxing Chen, name: Xiaodong Chen, name: Zhanchao Zhou, name: Tieyuan Chen, name: Yihong Zhuang, name: Guoshan Lu, name: Zenan Huang, name: Junbo Zhao, name: Lin Liu, name: Zhenzhong Lan, name: Bei Yu, name: Jianguo Li

The Mixture of Experts (MoE) architecture is a cornerstone of modern
state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate
scalability by enabling sparse parameter activation. However, traditional MoE
architecture uses homogeneous experts of a uniform size, activating a fixed
number of parameters irrespective of input complexity and thus limiting
computational efficiency. To overcome this limitation, we introduce Grove MoE,
a novel architecture incorporating experts of varying sizes, inspired by the
heterogeneous big.LITTLE CPU architecture. This architecture features novel
adjugate experts with a dynamic activation mechanism, enabling model capacity
expansion while maintaining manageable computational overhead. Building on this
architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs
developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model
during mid-training and post-training. GroveMoE models dynamically activate
3.14-3.28B parameters based on token complexity and achieve performance
comparable to SOTA open-source models of similar or even larger size.