Ziya2: Data-centric Learning is All LLMs Need
Authors: name: Ruyi Gan; name: Ziwei Wu; name: Renliang Sun; name: Junyu Lu; name: Xiaojun Wu; name: Dixiang Zhang; name: Kunhao Pan; name: Ping Yang; name: Qi Yang; name: Jiaxing Zhang; name: Yan Song
Published: 2023-11-06 17:49:34

Abstract: Various large language models (LLMs) have been proposed in recent years,
including closed- and open-source ones, continually setting new records on
multiple benchmarks. However, the development of LLMs still faces several
issues, such as high cost of training models from scratch, and continual
pre-training leading to catastrophic forgetting, etc. Although many such issues
are addressed along the line of research on LLMs, an important yet practical
limitation is that many studies overly pursue enlarging model sizes without
comprehensively analyzing and optimizing the use of pre-training data in their
learning process, as well as appropriate organization and leveraging of such
data in training LLMs under cost-effective settings. In this work, we propose
Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation
model, and further pre-trained on 700 billion tokens, where we focus on
pre-training techniques and use data-centric optimization to enhance the
learning process of Ziya2 on different stages. Experiments show that Ziya2
significantly outperforms other models in multiple benchmarks especially with
promising results compared to representative open-source ones. Ziya2 (Base) is
released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and
https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.