BaichuanSEED: Sharing the Potential of ExtensivE Data Collection and
  Deduplication by Introducing a Competitive Large Language Model Baseline
Authors: name: Guosheng Dong; name: Da Pan; name: Yiding Sun; name: Shusen Zhang; name: Zheng Liang; name: Xin Wu; name: Yanjun Shen; name: Fan Yang; name: Haoze Sun; name: Tianpeng Li; name: Mingan Lin; name: Jianhua Xu; name: Yufan Zhang; name: Xiaonan Nie; name: Lei Su; name: Bingning Wang; name: Wentao Zhang; name: Jiaxin Mao; name: Zenan Zhou; name: Weipeng Chen
Published: 2024-08-27 14:08:23

Abstract: The general capabilities of Large Language Models (LLM) highly rely on the
composition and selection on extensive pretraining datasets, treated as
commercial secrets by several institutions. To mitigate this issue, we
open-source the details of a universally applicable data processing pipeline
and validate its effectiveness and potential by introducing a competitive LLM
baseline. Specifically, the data processing pipeline consists of broad
collection to scale up and reweighting to improve quality. We then pretrain a
7B model BaichuanSEED with 3T tokens processed by our pipeline without any
deliberate downstream task-related optimization, followed by an easy but
effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency
and predictability throughout training and achieves comparable performance on
comprehensive benchmarks with several commercial advanced large language
models, such as Qwen1.5 and Llama3. We also conduct several heuristic
experiments to discuss the potential for further optimization of downstream
tasks, such as mathematics and coding.