Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts
  Language Models
Authors: name: Tianwen Wei; name: Bo Zhu; name: Liang Zhao; name: Cheng Cheng; name: Biye Li; name: Weiwei Lü; name: Peng Cheng; name: Jianhao Zhang; name: Xiaoyu Zhang; name: Liang Zeng; name: Xiaokun Wang; name: Yutuan Ma; name: Rui Hu; name: Shuicheng Yan; name: Han Fang; name: Yahui Zhou
Published: 2024-06-03 03:58:41

Abstract: In this technical report, we introduce the training methodologies implemented
in the development of Skywork-MoE, a high-performance mixture-of-experts (MoE)
large language model (LLM) with 146 billion parameters and 16 experts. It is
initialized from the pre-existing dense checkpoints of our Skywork-13B model.
We explore the comparative effectiveness of upcycling versus training from
scratch initializations. Our findings suggest that the choice between these two
approaches should consider both the performance of the existing dense
checkpoints and the MoE training budget. We highlight two innovative
techniques: gating logit normalization, which improves expert diversification,
and adaptive auxiliary loss coefficients, allowing for layer-specific
adjustment of auxiliary loss coefficients. Our experimental results validate
the effectiveness of these methods. Leveraging these techniques and insights,
we trained our upcycled Skywork-MoE on a condensed subset of our SkyPile
corpus. The evaluation results demonstrate that our model delivers strong
performance across a wide range of benchmarks.