Yuan 2.0-M32: Mixture of Experts with Attention Router
Authors: name: Shaohua Wu; name: Jiangang Luo; name: Xi Chen; name: Lingjun Li; name: Xudong Zhao; name: Tong Yu; name: Chao Wang; name: Yue Wang; name: Fei Wang; name: Weixu Qiao; name: Houbo He; name: Zeru Zhang; name: Zeyu Sun; name: Junxiong Mao; name: Chong Shen
Published: 2024-05-28 09:05:08

Abstract: Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a
mixture-of-experts architecture with 32 experts of which 2 experts are active.
A new router network, Attention Router, is proposed and adopted for a more
efficient selection of experts, which boosts the accuracy of 3.8% compared to
the model with classical router network. Yuan 2.0-M32 is trained with 2000B
tokens from scratch, and the training computation consumption is only 9.25% of
a dense model at the same parameter scale. Yuan 2.0-M32 demonstrates
competitive capability on coding, math, and various domains of expertise, with
only 3.7B active parameters of 40B in total, and 7.4 GFlops forward computation
per token, both of which are only 1/19 of Llama3-70B. Yuan 2.0-M32 surpass
Llama3-70B on MATH and ARC-Challenge benchmark, with accuracy of 55.89 and 95.8
respectively. The models and source codes of Yuan 2.0-M32 are released at
Github.