Model Merging in Pre-training of Large Language Models
Authors: name: Yunshui Li, name: Yiyuan Ma, name: Shen Yan, name: Chaoyi Zhang, name: Jing Liu, name: Jianqiao Lu, name: Ziwen Xu, name: Mengzhao Chen, name: Minrui Wang, name: Shiyi Zhan, name: Jin Ma, name: Xunhao Lai, name: Yao Luo, name: Xingyan Bin, name: Hongbin Ren, name: Mingji Han, name: Wenhao Hao, name: Bairen Yi, name: LingJun Liu, name: Bole Ma, name: Xiaoying Jia, name: Zhou Xun, name: Liang Xiang, name: Yonghui Wu

Model merging has emerged as a promising technique for enhancing large
language models, though its application in large-scale pre-training remains
relatively unexplored. In this paper, we present a comprehensive investigation
of model merging techniques during the pre-training process. Through extensive
experiments with both dense and Mixture-of-Experts (MoE) architectures ranging
from millions to over 100 billion parameters, we demonstrate that merging
checkpoints trained with constant learning rates not only achieves significant
performance improvements but also enables accurate prediction of annealing
behavior. These improvements lead to both more efficient model development and
significantly lower training costs. Our detailed ablation studies on merging
strategies and hyperparameters provide new insights into the underlying
mechanisms while uncovering novel applications. Through comprehensive
experimental analysis, we offer the open-source community practical
pre-training guidelines for effective model merging.