YuLan-Mini: An Open Data-efficient Language Model
Authors: name: Yiwen Hu; name: Huatong Song; name: Jia Deng; name: Jiapeng Wang; name: Jie Chen; name: Kun Zhou; name: Yutao Zhu; name: Jinhao Jiang; name: Zican Dong; name: Wayne Xin Zhao; name: Ji-Rong Wen
Published: 2024-12-23 17:47:53

Abstract: Effective pre-training of large language models (LLMs) has been challenging
due to the immense resource demands and the complexity of the technical
processes involved. This paper presents a detailed technical report on
YuLan-Mini, a highly capable base model with 2.42B parameters that achieves
top-tier performance among models of similar parameter scale. Our pre-training
approach focuses on enhancing training efficacy through three key technical
contributions: an elaborate data pipeline combines data cleaning with data
schedule strategies, a robust optimization method to mitigate training
instability, and an effective annealing approach that incorporates targeted
data selection and long context training. Remarkably, YuLan-Mini, trained on
1.08T tokens, achieves performance comparable to industry-leading models that
require significantly more data. To facilitate reproduction, we release the
full details of the data composition for each training phase. Project details
can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.