Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated
  Parameters by Tencent
Authors: name: Xingwu Sun; name: Yanfeng Chen; name: Yiqing Huang; name: Ruobing Xie; name: Jiaqi Zhu; name: Kai Zhang; name: Shuaipeng Li; name: Zhen Yang; name: Jonny Han; name: Xiaobo Shu; name: Jiahao Bu; name: Zhongzhi Chen; name: Xuemeng Huang; name: Fengzong Lian; name: Saiyong Yang; name: Jianfeng Yan; name: Yuyuan Zeng; name: Xiaoqin Ren; name: Chao Yu; name: Lulu Wu; name: Yue Mao; name: Tao Yang; name: Suncong Zheng; name: Kan Wu; name: Dian Jiao; name: Jinbao Xue; name: Xipeng Zhang; name: Decheng Wu; name: Kai Liu; name: Dengpeng Wu; name: Guanghui Xu; name: Shaohua Chen; name: Shuang Chen; name: Xiao Feng; name: Yigeng Hong; name: Junqiang Zheng; name: Chengcheng Xu; name: Zongwei Li; name: Xiong Kuang; name: Jianglu Hu; name: Yiqi Chen; name: Yuchi Deng; name: Guiyang Li; name: Ao Liu; name: Chenchen Zhang; name: Shihui Hu; name: Zilong Zhao; name: Zifan Wu; name: Yao Ding; name: Weichao Wang; name: Han Liu; name: Roberts Wang; name: Hao Fei; name: Peijie She; name: Ze Zhao; name: Xun Cao; name: Hai Wang; name: Fusheng Xiang; name: Mengyuan Huang; name: Zhiyuan Xiong; name: Bin Hu; name: Xuebin Hou; name: Lei Jiang; name: Jiajia Wu; name: Yaping Deng; name: Yi Shen; name: Qian Wang; name: Weijie Liu; name: Jie Liu; name: Meng Chen; name: Liang Dong; name: Weiwen Jia; name: Hu Chen; name: Feifei Liu; name: Rui Yuan; name: Huilin Xu; name: Zhenxiang Yan; name: Tengfei Cao; name: Zhichao Hu; name: Xinhua Feng; name: Dong Du; name: Tinghao She; name: Yangyu Tao; name: Feng Zhang; name: Jianchen Zhu; name: Chengzhong Xu; name: Xirui Li; name: Chong Zha; name: Wen Ouyang; name: Yinben Xia; name: Xiang Li; name: Zekun He; name: Rongpeng Chen; name: Jiawei Song; name: Ruibin Chen; name: Fan Jiang; name: Chongqing Zhao; name: Bo Wang; name: Hao Gong; name: Rong Gan; name: Winston Hu; name: Zhanhui Kang; name: Yong Yang; name: Yuhong Liu; name: Di Wang; name: Jie Jiang
Published: 2024-11-04 16:56:26

Abstract: In this paper, we introduce Hunyuan-Large, which is currently the largest
open-source Transformer-based mixture of experts model, with a total of 389
billion parameters and 52 billion activation parameters, capable of handling up
to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior
performance across various benchmarks including language understanding and
generation, logical reasoning, mathematical problem-solving, coding,
long-context, and aggregated tasks, where it outperforms LLama3.1-70B and
exhibits comparable performance when compared to the significantly larger
LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale
synthetic data that is orders larger than in previous literature, a mixed
expert routing strategy, a key-value cache compression technique, and an
expert-specific learning rate strategy. Additionally, we also investigate the
scaling laws and learning rate schedule of mixture of experts models, providing
valuable insights and guidances for future model development and optimization.
The code and checkpoints of Hunyuan-Large are released to facilitate future
innovations and applications.
  Codes: https://github.com/Tencent/Hunyuan-Large
  Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large