Pangu Pro MoE: Mixture of Grouped Experts for Efficient Sparsity
Authors: name: Yehui Tang, name: Xiaosong Li, name: Fangcheng Liu, name: Wei Guo, name: Hang Zhou, name: Yaoyuan Wang, name: Kai Han, name: Xianzhi Yu, name: Jinpeng Li, name: Hui Zang, name: Fei Mi, name: Xiaojun Meng, name: Zhicheng Liu, name: Hanting Chen, name: Binfan Zheng, name: Can Chen, name: Youliang Yan, name: Ruiming Tang, name: Peifeng Qin, name: Xinghao Chen, name: Dacheng Tao, name: Yunhe Wang

The surgence of Mixture of Experts (MoE) in Large Language Models promises a
small price of execution cost for a much larger model parameter count and
learning capacity, because only a small fraction of parameters are activated
for each input token. However, it is commonly observed that some experts are
activated far more often than others, leading to system inefficiency when
running the experts on different devices in parallel. Therefore, we introduce
Mixture of Grouped Experts (MoGE), which groups the experts during selection
and balances the expert workload better than MoE in nature. It constrains
tokens to activate an equal number of experts within each predefined expert
group. When a model execution is distributed on multiple devices, this
architectural design ensures a balanced computational load across devices,
significantly enhancing throughput, particularly for the inference phase.
Further, we build Pangu Pro MoE on Ascend NPUs, a sparse model based on MoGE
with 72 billion total parameters, 16 billion of which are activated for each
token. The configuration of Pangu Pro MoE is optimized for Ascend 300I Duo and
800I A2 through extensive system simulation studies. Our experiments indicate
that MoGE indeed leads to better expert load balancing and more efficient
execution for both model training and inference on Ascend NPUs. The inference
performance of Pangu Pro MoE achieves 1148 tokens/s per card and can be further
improved to 1528 tokens/s per card by speculative acceleration, outperforming
comparable 32B and 72B Dense models. Furthermore, we achieve an excellent
cost-to-performance ratio for model inference on Ascend 300I Duo. Our studies
show that Ascend NPUs are capable of training Pangu Pro MoE with massive
parallelization to make it a leading model within the sub-100B total parameter
class, outperforming prominent open-source models like GLM-Z1-32B and
Qwen3-32B.