Model Merging in Pre-training of Large Language Models
Authors: Yunshui Li
Yiyuan Ma
Shen Yan
Chaoyi Zhang
Jing Liu
Jianqiao Lu
Ziwen Xu
Mengzhao Chen
Minrui Wang
Shiyi Zhan
Jin Ma
Xunhao Lai
Yao Luo
Xingyan Bin
Hongbin Ren
Mingji Han
Wenhao Hao
Bairen Yi
LingJun Liu
Bole Ma
Xiaoying Jia
zhou Xun
liang xiang
Yonghui Wu
Venue: NeurIPS 2025 poster ()
Keywords: Large language model; Pretrain; Model merge; Stable training;
Primary area: deep_learning

Abstract: Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unexplored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experts (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies on merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel applications. Through comprehensive experimental analysis, we offer the open-source community practical pre-training guidelines for effective model merging.

Model Merging in Pre-training of Large Language Models
Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu Mengzhao Chen, Minrui Wang, Shiyi Zhan, Jin Ma, Xunhao Lai, Yao Luo, Xingyan Bin Hongbin Ren, Mingji Han, Wenhao Hao, Bairen Yi, Lingjun Liu, Bole Ma, Xiaoying Jia Xun Zhou, Liang Xiang, Yonghui Wu
ByteDance Seed
1 liyunshui@byt wwwpunsssiyyytednnc..cmmm
Abstract
Model merging has emerged as a promising technique for enhancing large language models, though its application in large-scale pre-training remains relatively unex- plored. In this paper, we present a comprehensive investigation of model merging techniques during the pre-training process. Through extensive experiments with both dense and Mixture-of-Experte (MoE) architectures ranging from millions to over 100 billion parameters, we demonstrate that merging checkpoints trained with constant learning rates not only achieves significant performance improvements but also enables accurate prediction of annealing behavior. These improvements lead to both more efficient model development and significantly lower training costs. Our detailed ablation studies merging strategies and hyperparameters provide new insights into the underlying mechanisms while uncovering novel appli- cations. Through comprehensive experimental analysis, offer the open-source community practical pre-training guidelines for effective model merging.
Introduction
Modern large language models (LLMs) [Seed et al., 2025 Achiam et al.) 2023 Guo et al. 2025 Team al. 20244] Yang et al., [2024a] have demonstrated remarkable capabilities with widespread applications across diverse tasks. Despite their exceptional performance in fundamental tasks, LLMs still face several critical challenges, including the extensive pre-training costs, discounted effectiveness of domain-specific post-training. imprecisely--predittabll performance scaling, well as the instability of large-scale training. Model merging (Yang et al. 22024). as a relatively young topic, presents promising approach to alleviate these practical challenges.
Recently, the benefits of model merging have been primarily studied in the post-training stage, where several models fine-tuned on different downstream tasks are combined into a single but more versatile model [Ilharco et al., 2023] Zhou et al. 2024 Yu al.) (022).. For example, using the DARE YYu et al. [024]] method to merge Wizard . Xu et al.) [0224] with WizardMath [Luo] al. 2025] shows a significant performance enhancement on GSM8K [Cobbe et al. 202T], raising its score from 2.2 to 66.3, In contrast, research on model merging during the pre-training phase remains scarce. Such pre-training merging typically involves combining checkpoints from a single training trajectory, explored in LAWA [Kaddour] 2022 which utilizes model merging to accelerate the LLM training. However, as the model and data scales dramatically, independent researchers struggle to evaluate model merging's impact large-scale models, mainly due limited access to intermediate checkpoints from extensive pre- training.. Although DeepSeek (Liu et al.) [024a]] and LLaMA-3 (Grattafiori al., 20244 have both indicated their employment of model merging techniques for model development, detailed information regarding these techniques has not been publicly disclosed.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we mainly focus model merging during the pre-training stage, introducing Pre- trained Model Average (PMA), a novel strategy for model-level weight merging during pre-training. To comprehensively evaluate PMA, we trained a diverse set of LLMs of varying sizes and architectures from scratch, including Dense Grattaaiia et al. [2024] with parameters spanning from 41 1M to 70B, as well as Mfxture-off-xperre (MoE) architectures Shazeer et al. [2017] with activated/total parameters ranging from 0.7B/7B to 20B/200B. We first investigate the performance impact of PMA and establish systematic evaluations across different phases of the warmup-stable-decay (WSD) learning schedule, which lately becomes popular choice of Ir scheduler for LLM pre-training since Hu et al. [202]]. Experimental results demonstrate that model merging during the stable training phase yields consistent performance gains at different training steps. More remarkably, applying PMA at early-stage of the cosine-decay phase usually achieve comparable or even superior performance to their final-stage annealing counterparts. These findings suggest that during the extensively lengthy pre-training stage with constant Ir, PMA can serve fast, reliable yet low- cost simulator for the annealed performance, enabling both faster validation cycles and significant computational savings.
Building upon our PMA framework, we first evaluate its performance with various prevalent merging strategies, including Simple Moving Average (SMA) Johnston et al. [1999]. Weighted Moving Average (WMA) Perry [2000] and Exponential Moving Average (EMA) Hunter [1986]. Notably, our experiments demonstrate that the performance differences among these methods gradually become negligible. We further investigate how these important factors PMA, namely, the interval between each merging checkpoint, the number of models involved in merging, and the size of the model, would affect merging performance. Our analysis reveals two important findings: First, the optimal merging interval exhibits clear scaling relationship with model size. Second, incorporating more checkpoints in the merging process consistently improves performance once training is completed.
Furthermore, we also investigated whether PMA could produce more effective initialization weights for the consecutive continued training (CT) or supervised fine-tuning (SFT) Wei et al. [2022] stages to enhance the downstream model performance. We practically observed that entering CT and SFT stages with PMA applied could yield smoother GradNorm curves, which thus helps stabilize the training dynamics yet without harming the performance, compared to initializing these stages with the latest available checkpoint as usual. This finding inspire a novel application of model merging for training stabilization, which we dubbed PMA-init. We demonstrate that in scenarios when the LLM training experiences severe irrecoverable loss spikes with broken training dynamics, applying PMA-init over N preceding checkpoints to resume training, enables reliable recovery from unstable training trajectories.
In summary, paper makes the following key contributions:
We present the Pre-trained Model Averaging (PMA) strategy, a novel framework for model merging during LLM pre-training. Through extensive experiments across model scales (from millions to over 100B parameters), we demonstrate that merging checkpoints from the stable training phase produces consistent and significant performance improvements.
We delved into novel applications of model merging for weight initialization (PMA-init), to help stabilize training process without harming the downstream performance, especially when it suffers from irrecoverable loss spikes with broken training dynamics. Through extensive experiments, we demonstrate the effectiveness of PMA-init on both CT and SFT stages.
We also comprehensively ablated various model merging techniques with their associated hyper-parameters. Our findings offer the research community practical pre-training guide- lines with effective model merging. Nevertheless, the low cost and rapid deployment of PMA also make it reliable and economic monitor for the pre-training process, to flexibly simulate the ultimate model performance after annealing.
2 Related Work
Model merging is emerging field undergoing rapid development, with diverse applications across various domains. Typically, model merging is implemented during the post-training phase [Ihharco]
al. 2023 Zhou et al. 2024 Yu et al. 2024 Yadav et al. 2024). where multiple models fine-tuned
2

---

7M
Itunanecol
MMLU
COMMIK
Tharaneral
MMLU
GRMMK
flumneeaa
MMILI
GSASAK
Ilaraneeal
MMLUU
GSMIK
MoT-L.58/138
MoR-38/200
Mol--1R/1008
Malf-208/2008
Figure I: Comparison of downstream task performance for MoE models of varying sizes under stable training. before and after model merging.
on different downstream tasks are combined by merging their weights. This process effectively integrates the distinct capabilities of each individual model, resulting in a unified model that exhibits robust and comprehensive performance.
Recently, several methods have advanced this field significantly. For instance, Task Arithmetic (II
harco et al. 2023), Ties-Merging (Yadav et al. 2023). and AdaMerging [Yang et al. 2024c integrate Vision Transtormer (ViT) models [Dosovitskiy al. 202T trained on distinct visual classification tasks, producing single model capable of task object classification. PAPA Dolicoeur-Maiin Martineau
al. 2023] integrates the broad applicability of ensembling with the computational efficiency of weight averaging. MetaGPT [Zhou et al., 2024) frames model merging as multi-task learning problem, aiming to minimize the average loss hetween the merged model and individual task-specific
models. Fisher Merging [Matena and Raffel 2022] employs a weighted fusion of model parameters,
with weights determined by the Fisher information matrix. RegMean [Jin et al., [023] elegantly
addresses the merging process by formulating it linear regression problem solvable through closed-form solutions. modlutionnrymmooll-enee coefficients using evolutionary algorithms. Additionally, DARE Yu et al. 2024] merges multiple task-specific language models into a versatile unified model by randomly dropping and subsequently rescaling the delta parameters.
[Akiba et al. 2025 efficiently optimizes merging
However, research on model merging during the pre-training phase remains relatively limited. Such studies typically refer to incorporating checkpoints within a single training trajectory during large language model (L.LM) pre-training. For example, LAWAA [Kaddour] 2022 Hägele al. 20244] [i] al. 2022 demonstrated that merging checkpoints intermediate stages can significantly accelerate training. Sanyal et al. [Sanyal et al. 2024] further indicated that checkpoint averaging combined with high learning rate in pre- praiiigg trajectories contributes to faster convergence. Additionally, Checkpoint Merging [Liu et al. 2024b provided comprehensive evaluation the effectiveness of merging checkpoints at different stages during the pre-training of the Baichuan2 (Yang al. 2023 LLM. Furthermore, technical reports of large-scale models such as Deepseek V3 [Liu et al. 2024a] and LLaMA3.1 [Grattafiori et al. 2024] also mention the of model merging techniques during pre-training. although detailed methodologies have not been publicly disclosed. This paper primarily explores techniques for model merging within the pre-training paradigm. To the best of our knowledge, this is the first study to provide detailed technical insights into scaling model merging methods to significantly larger model sizes. We also discuss practical approaches for effective model merging and analyze its potential capabilities as well as its limitations.
3 Preliminaries
In this section, describe the fundamental experimental framework, introduce the notations and concepts used model merging, and present multiple variants of model merging techniques.
Experimental setup. In terms of model architecture, we independently trained a series of MoE and dense models. We employ a Warmup-Stable-Deaay (WSD) learning rate scheduler Hu al. [2024), which begins with short warmup period, followed by an extended period of stable training constant learning rate, and concludes with annealing to relatively small learning rate. The learning rates are determined according to scaling law guidelines Bi al. 2024
Kaplan et al. 2020

---

employing optimal values for training an internal pretraining corpus comprising trillions of tokens. Although specific model architectures and datasets have not yet been publicly released. we posit that our findings are not strongly tied to these particular choices, as subsequent experiments primarily focus on MoE structures. Related conclusions for dense models are provided in the Appendix [ For evaluation, we primarily report results on open-source benchmarks in both few-shot and zero-shot
settings, including: ARC-Challenge Clark et al. (2018), . BBH Suzgun et al. (2023), DROP Dua et al.
[2019]. WinoGrande Sakaguchi et al. [202]] HellaS wag Zellers et al. [2019] MMLU Hendrycks
et al. 202T], C-Eval C-EvalHuang et al. [2023], TriviaQA Joshi et al. 2017]. Ape2 TOK Zhhoo et al. [2020].
GSM8K Cobbe et al. [2021], HAATH/haa
Chen 2021], AGIEval Zhong et al. [024]]. GPQA Rein et al. 2024). and MMLU- Wang et al.
al. (020 MBPP Austin al. [202], HumanEval Chen
et al.
2024 The weighted average score across these benchmarks serves as the model's 's comprehensive
performance metric. Unless otherwise specified, we report this score as the model's performance
metric to ensure evaluation reliability.
Notions and concepts. Our main focus is on model merging during pre-training, where the merged entities are sequential checkpoints along the training trajectory. Suppose we aim to merge N models, with each model's s parameters denoted as M, (where i ranges from to N). Each model has an associated weighting coefficient wi, and the merged model Mang is computed as:
N
Mavg
wo W.
We assume that the data consumption of these models form an arithmetic sequence with a common difference V, formulated as:
V T++1 - T,
where T; represents the cumulative number of tokens consumed by the i-th model.
Model merging variants. Model merging techniques vary primarily in how they assign weights (w; to individual models. This paper examines three popular approaches for weight assignment, namely the Simple Moving Average (SMA), Exponential Moving Average (EMA), and Weighted Moving Average (WMA).
The first approach, Simple Moving Average (SMA), treats all models equally. For instance, when combining 10 models, each model is assigned a weight of w; = 0.1. The SMA is formulated as:
N
Mavg
M,
The second approach, Exponential Moving Average (EMA), emphasizes later models by assigning weights that decay exponentially, making EMA more sensitive to recent changes. The EMA is expressed recursively as:
M avg
a M, (1-a) aug
i i [2,
Here, a, the smoothing factor (typically between 0 and 1), controls the balance between the current model M, and the previous EMA result M (1-1)
The third approach, Weighted Moving Average (WMA), also prioritizes recent models but uses distinct weighting scheme. In WMA, each model is assigned specific weight, often increasing linearly for later models (e.g., Wc i). The weighted sum then normalized to compute the average, formulated as follows:
w/s
Mavg
M,
Usu
W
(5)
Wsum
These methods offer flexible ways to combine models based on their recency and relevance. Choosing the right approach depends the specific application and desired emphasis newer data.
4 Experiments
In this section, we delve into the experimental core of our study, systematically addressing six critical questions surrounding model merging in the context of pre-training: 1) How does model merging

---

affect performance? 2) How do different merging methods affect final performance? 3) How to determine the optimal interval and number of weights merge for various model sizes? Do merged pre-trained models contribute to better downstream training? 5) Does model merging improve the stability of training? 6) What processes unfold during model merging? Through these experiments, we aim to provide comprehensive insights into model merging, offering practical guidance for its application and shedding light on its theoretical underpinnings.
4.1 How does model merging affect model performance?
Current learning rate schedule methods mainly involve constant learning rates or cosine annealing. In model pre-training, employed the Warmup-Stable-DDcaa (WSD) strategy Hu al. (2024). which combines constant learning rate phase with a subsequent cosine decay phase Loshchilov and Hutter [0017]. To explore the effects of model merging under different learning rate schedules, conducted experiments during both constant learning rate phase and cosine dacay phase.
In the constant learning rate phase, merged fully trained models of various sizes. As shown in Figure [ the merged models exhibited significant performance improvements across multiple downstream tasks. For example, the Humaneval benchmark, Seed-MoE-1.3B/133 improved from 31.1 to 36.6 points, and Seed-MoF- 1aooe---//1000 10B/100B increased from 54.3 to 61.6 points. While larger models showed less pronounced gains certain benchmarks, such BBH, this was likely due the near-saturation of these metrics. Overall, the improvements were robust and consistent across model sizes.
Next, we performed model merging in the cosine annealing phase by collecting weights from the annealing stages of Seed-MoE-1.3B/11
Beed-MoE-10B//ON0 and Seed-MoE- 15MM/--55B//0BB As depicted in Figure 2 as the learning rate gradually decreased, the models converged steadily. with performance continuing to improve. Interestingly, the early annealing stage, the results of PMA were comparable to those at the end of the annealing process. In some cases, particularly for larger models, the merged models surpassed those naturally annealed.
Annealing
Annealing
BST
MuF-1.3R/138
MoF-1oR/1008
MoB-15R/1508
Figure 2: Comparison of overall performance for MoE models of varying sizes under annealing training, before and after model merging. The learning rate follows cosine schedule during the annealing process. The x- x-axis shows the count of training tokens.
These findings raised a question: could we simplify the training process by using only the Warmup- Stable phases alongside PMA, skipping the decay phase, and avoiding learning rate adjustments? To investigate, we forked two training runs from the stable phase Sced-MoE-L133//1
at 1.4T tokens. One continued with constant learning rate, while another underwent anncaling, each training for an additional 250B tokens. We then merged the models trained with the constant lcarning rate. As shown in Figure early in training, the merged models significantly outperformed both the constant learning rate and annealed models. Even later, their performance was comparable to the annealed models.
This suggests that pre-training with constant learning rate, combined with model merging, can effectively match the performance of an annealed model at any point in the training process without the need for learning rate annealing. This approach accelerates model validation and significantly reduces computational resource demands.

---

PMA
Asnealing
Contont
Portomsnns
60%
1:811
1.51
1.511
347
1.411
1,51
1.511
1.41
1.41
1.281
1,51
1.31
1.41
1.481
1.51
101
Humaneral
BB11
MMLU
GSMBX
Figure 3: Comparison of downstream task performance between model merging results under stable training and the real annealed model. The x-axis shows the count of training tokens.
Baseline
WMA
SMA
EMA GHOLE
EMA gH022
00.51
MEAT
565
56.15
35.0
50
Performance
$1.17
$0.12
$7.00
40
05.53
204B
452B
1112B
1607B
Figure 4: Impact of different model merging methods final model performance.
4.2 How do different merging methods affect final performance?
In this section, systematically evaluate how different merging strategies affect the performance of merged models. Specifically, we focus on three distinct approaches: EMA. WMA, and SMA. The EMA method employs exponentially decaying weights w; a(1 - N giving higher importance to more recent checkpoints. WMA assigns linearly increasing weights w; = i, also prioritizing more recent checkpoints. In contrast, SMA applies uniform weighting, treating all checkpoints equally regardless of their position in the training sequence.
We conducted experiments on Seed-MoE- 1.3/13B and showed the results in Figure At 204B training tokens, all merging methods enhanced model performance compared to the pre-merged model, but WMA delivered the best results. This suggests that in the early phases of training, when model weights undergo significant changes, assigning higher weights to checkpoints with more training tokens produces superior models. This is further supported by the fact that EMA,=0.2 outperforms EMA,=0.1- However, as training advances to later stages and model weights stabilize, the performance differences between merging methods diminish. For its simplicity and stability, we primarily use SMA for model merging in subsequent experiments.
4.3 How to determine the optimal interval and number of weights to merge for various model sizes?
Beyond the merging technique itself, two other factors may also affect the effectiveness of model merging: the interval V between selected models and the number of models N. We performed ablation studies on the Mo-d-MME--1.3/133 model to investigate these effects, starting with the impact of the interval. As illustrated in the upper part of Figure we fixed N 10 and tested intervals of V 4B, 8B, 16B, and 32B. Notably, at 204B with V 32B, we reduced N to 6 due to insufficient models. In the early stage of training, at 204B tokens, merged results with V 16B and V 32B underperformed the baseline. This is likely because large intervals incorporated unstable weights from the initial training phase, leading to significant weight disparities and suboptimal outcomes. As training progressed and weights stabilized, the performance gap across different V settings gradually narrowed. In practice, the optimal interval scales with model size, following these observed patterns: interval of around 8B tokens for 1.3B/13B models, 4B tokens for 0.7B/7B models,

---

and approximately 80B tokens for 10B/100B models. This aligns with the tendency of larger models use larger batch sizes McCandlish et [2118)
Next, we set V 8B and explored how the number of merged models N affects performance, testing N 3, 6, 10, and 13. As shown in the lower part of Figure early in training, incorporating more models introduced unstable weights, which reduced the performance of merged models. However, once training was complete, merging a larger number of models led significant performance improvements. Notably, the overall performance for N = 3 was nearly point lower than for N 15. To strike a balance between computational cost and performance gains, we opted for 10 in further experiments.
N
Baseline
V-6BB
V-16B
Performance
S4.
533
Tokee
204B
152B
1112B
1607B
36.1
14.2
Peformance
542
Tokes
2048
452B
11128
1607B
Figure 5: Impact of different model merging hyper-parameters on final model performance of the Seed-MoE-L.3/13 Model.
4.4 Do merged pre-trained models contribute to better downstream training?
A complete LLM training process typically involves multiple stages, which pretraining, continual training (CT), supervised fine-tuning (SFT) and reinforcement learning (RL) in sequence. In light of the capacity of PMA to improve pretraining performance, we conjecture that merged pretrained models may similarly prove beneficial for downstream stages. To verify this hypothesis, we initialized downstream training with PMA, which we dubbed as PMA-init, and investigated its impacts the baselines (which are initialized from their original checkpoints) for both CT and SFT stages.
066
0.4
15
064
0.43
1.0
044
3500
2000
7200
.000
13:00
15000
1700
2000
0.59
Training Stop
Carouned Tolkane
Figure 6: Comparisons of loss curves (left) and performance metrics (right) during CT stage with varying ir schedules, where cosine scheduler adopted decay learning rate from (peak Irend (denoted as Ir (rpak - Irend). PMA and baseline, stand for whether our PMA-in technique is employed or not, respectively.

---

CT stage. We first conducted an ablation study to assess the sensitivity of the PMA-init of the CT stage with varying learning rate schedules. Specifically, we experimented with Sced-MoE-O.7B/7B models merged after stable training on approximately 1 trillion tokens. As illustrated in Figure (left). the initialization weights obtained via PMA consistently achieved marginally lower loss at the initial training phase, against the baseline with the same training configuration. As training progresses, the loss values for models with different initialization weights converge to comparable levels. It's worth noting that in the loss curve, the purple line significantly overlaps with the blue line, and the brown line significantly overlaps with the pink line. Another observation is made in the Figure 6 (right), where evaluation on the MMLU benchmark reveals that the PMA-init models outperform the baseline early in training. While these models tend to retain a slight performance edge in later stages, their results other tasks may be slightly suboptimal, leading to overall performance parity with the baseline. Experiments across varied learning rate schedules corroborate these findings, indicating that models converge to similar performance levels by the end of training, and extensive learning rate tuning is required for PMA-init.
SFT stage. We next analyzed the impact of PMA-init on the SFT stage, where the detailed results can be found in the Appendix B Although initialization with merged weights occasionally yields performance improvements, such gains are not consistently observed. Nonetheless, this approach does not adversely affect downstream training outcomes and may be a viable strategy for researchers seeking to enhance model performance.
4.5 Does model merging improve the training stability?
In large-scale LLM training, infrastructure issues are almost inevitable and often lead to training instability phenomena such as loss spikes diverging. Specifically, loss spike occurs when, at a specific point during the multi-stage training, the model's predictions deteriorate significantly compared to previous iterations. This phenomenon is often observed alongside gradient norm (GradNorm) explosion during backpropagation, which causes large weight updates and eventually lead to a irrecoverable spike in its loss function Cohen et al. [202T). In the experiments detailed in Section 4.4, as illustrated in Figure (7 (left), we observed that a model initialized with PMA- -iiit t for SFT stage demonstrated a notably more stable GradNorm metric compared the baseline. This stability is also evident in the reduced frequency of loss spikes relative to the baseline. Since applying PMA-init for downstream training does not impact the model's 's final performance and remains robust across different learning rates, we established a series of experiments to explore whether model merging could enhance training stability.
Given the extremely high expenses associated, it is unfeasible to conduct a direct analysis of training instability in LLM pre-training. Experiments Wortsman al. (2024) show that small models using relatively large learning rate will exhibit unstable training characteristics similar to those of large models. We thus reproduce the instability phenomena on small models to study the influence of our PMA-init on training stability. In one such experiment, we trained a 330M/3.33 3B MoE model from scratch using an exceptionally high learning rate of 6e-3. As shown in Figure 7 (right), the model overshot the optimal weights, resulting in unstable training and abrupt loss spikes as expected, and was irreversible to its original trajectory. To address this, we adopted PMA-init with three checkpoints saved before the training collapse happened, to resume the pre-training process. As depicted by the red line in Figure 7 (right), the resumed training process stabilized, successfully navigating past the point of the loss spike and continuing along its original training trajectory.
These results highlight that PMA-init can reliably enhance the multi-stage training stability. When a loss spike occurs, one can merge the model checkpoints from before the spike and resume training from that point. This approach provides an alternative solution to avoid retraining the model from scratch, thereby substantially reducing the waste of computational resources.
4.6 Investigating the Mechanisms of Model Merging
To gain deeper insight into the underlying mechanisms that enable model merging to be effective, we provide both qualitative and quantitative analyses, employing mathematical derivations and visualizations of weight distributions. Due to space limitations, this section focuses solely on the qualitative analysis. For the guantitative analysis, please refer to Appendix C for more details.
8

---

23
IMAIR
PMA Inl
1a0
1.0
300
370
500
610
700
Traininn
Thininn
Figure 7: Left:: GradNorm comparisons for SFT training initialized with PMA- init. Right: Compari- son of pre-training loss curves between resuming with MMit and the original training.
HogelMall
75
Weighe22
6.0
50
4.5
Weight-1
Figure 8: Visualization MMLU score contour lines, comparing the weights an original model with those of merged model. Black dots represent the parameter locations of various individual model checkpoints.
We selected several checkpoints from the pre-training of Seed-MoE-1 M.oE-1.3B/13BB and visualized the average distribution two selected parameters from specific layer. Using these points, we generated contour lines for MMLU scores, as illustrated in Figure 8 The weight positions of various individual models are marked as black dots. These dots are distributed along the MMLUU score contours, revealing a discernible "complementary" pattern. The averaged weight position (representative of the merged model) is often situated closer region higher MMLU scores (a better optimum) than many individual model checkpoints. This visualization also provides an intuitive explanation for why model merging yields diminished improvements when models are annealed to very low learning rate: at such stage, the models to be merged are already tightly converged within specific local optimum. Merging them essentially averages points within this already narrow basin, making unlikely to escape to significantly better different optimal region.
5
Conclusion
This research pioneers deeper exploration of model merging within the challenging pre-training stage of large-scale models. By training a spectrum of MoE and Dense models and performing rigorous ablations, we established that merging checkpoints from stable training phases not only yields significant performance gains and predicts annealing but also streamlines development and reduces costs. Our work provides concrete guidance on merging strategies, optimal parameters, and downstream applications, alongside insights into the underlying mechanisms. These contributions equip the open-source community with the knowledge and tools for more efficient model development through pre-training merging.

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, pages 1-10, 2025.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprini arXiv: 108.07732. 2021.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek IIm: Scaling open-source language models with longtermism. arXiv preprini arXiv:2401.000 .0294,4 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803 050550 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages
2019.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501.12944 2025.
Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, Martin Jaggi, et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232-76264, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia Guoyang Zeng, da- hai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In First Conference on Language Modeling, 2024.
10

---

Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang. Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, jiayi lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
J Stuart Hunter. The exponentially weighted moving average. Journal of quality technology, 18(4): 203-210, 1986.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations, 2023.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The International Conference on Learning Representa- tions, 2023.
FR Johnston, John E Boyland, Maureen Meadows, and E Shale. Some properties of a simple moving average when applied to forecasting a time series. Journal of the Operational Research Society, 50 (12):1267-1271, 1999.
Alexia
Emy Gervais, Kilian Fatras, Yan Zhang, and Simon Lacoste-Julien. Population parameter averaging (papa). arXiv preprinl arXiv:334.003094 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors. Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1601-1611, 2017.
Jean Kaddour. Stop wasting my time! saving days of imagenet and BERT training with latest weight averaging. In Advances in Neural Information Processing Systems Workshop, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXi::2001.0836/ 2020.
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging: Efficient training by optimizing historical solutions. In The Eleventh International Conference on Learning Representations, 2022.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437. 2024a.
Deyuan Liu, Zecheng Wang, Bingning Wang, Weipeng Chen, Chunshan Li, Zhiying Tu, Dianhui Chu, Bo Li, and Dianbo Sui. Checkpoint merging bayesian optimization in Ilm pretraining. arXiv preprint arXiv:2403. 19390, 2024b.
Ilya Loshchilov and Frank Hutter. SGDR:
Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. In International Conference on Learning Representations, 2025.
Michael S Matena and Colin A Raffel. Merging models with weighe- averaging. Advances in Neural Information Processing Systems, 35:17703-17716 2022.
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training. arXiv preprint / 2018
Marcus Perry. The weighted moving average technique. Wiley Encyclopedia of Operations
Research and Management Science, 2010.
11

---

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-leve grogle-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, and sujay sanghavi. Early weight averaging meets high learning rates for LLM pre-training. In First Conference Language Modeling, 2024.
ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, et al. Seed-thinking-vi. 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504. 13914, 2025.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Noam Shazeer, * Azalia Mirhoseini, *Kzyysttff Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mffture-of-experts layer. In International Conference on Learning Representations, 2017.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung. Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics, pages 13003- 13051, 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. In The Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: challenging, contanination-liiieed LLM benchmark. In International Conference on Learning Representutions, 2025.
Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie E Everett, Alexander A Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-Dickstein, Kelvin Xu, Jachoon Lee, Justin Gilmer, and Simon Korbblith. Small-scale proxies for turge-scale transformer training instabilities. In International Conference on Learning Representations, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin. and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The International Conference on Learning Representations, 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging Resolving interference when merging models. In Conference on Neural Information Processing Systems, 2023.
12

---

Prateek Yadav, Tu Vu, Jonathan Lai. Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint arXiv:2410.03617, 2024.
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305. 2023.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.00 07666, 2024b.
Enneng Yang. Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference Learning Representations, 2024c.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models free lunch. In International Conference Machine Learning, 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 14476, 2025.
Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTiaa Fan, Zhengyin Du, Xiangpeng Wei, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint 2rXir:2504.05/188 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine 2019.
really finish your sentence? arXiv preprint arXiv:1905.077
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang. and Jingming Liu. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009. 11506, 2020.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Findings of the Association for Computational Linguistics: NAACL, pages 2299 2314, 2024.
Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. MetaGPT: Merging large language models using model exclusive task arithmetic. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen, editors, Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1711-1724, 2024.
13

---

A The Effect of Model Merging in Dense Models
We also conducted model merging experiments Dense architecture models, which mirror the exact architecture of LLaMA 3.1 Grattafiori et al. 2024], ranging from small Dense-41 IM models to large Dense-70B models. Since the 41IM and 2B models were not sufficiently trained, used a configuration of N=6 for merging, with weight intervals (V) of 2B and 5B tokens, respectively. For the 8B and 70B models, which were trained more thoroughly, we used N=10, with V values of 15B and 40B for merging. As shown in Figure models of different sizes achieved significant improvements on downstream tasks after model merging. Notably, the performance gains of larger models were not smaller than those of smaller models. Specifically, Dense-70B improved from 50.6 to 57.9 on Humaneval and from 85.9 91.3 .3 on GSM8K. This further validates the robustness and generalization ability of PMA, demonstrating that it can work across different model architectures and sizes.
Bastine
J54
Paroomanee
Humaneval
B3H
MNLU
GSMeK
Hummesal
BSH
MNLU
GSMDK
Humanvall
BRH
MNLU
GSNeK
Humaesal
BBH
MMLU
GSMDK
Desse
Dense
Figure 9: Comparison of downstream task performance for dense models of varying sizes under stable training, before and after model merging.
B Model Merging at the CT Stage for Supervised Fine-Tuning
PMA-init can be integrated during the CPT annealing phase, where PMA weights are merged to provide strong starting point for fine-tuning. We conducted an ablation study to assess the sensitivity of the PMA-init during the SFT stage to varying learning rate schedules. This study included experiments on merged MoE-15B/1K
models following stable training approximately 16T tokens, as well as after further training on tokens with cosine annealing. We conducted SFT training for 220M tokens using both the original weights and PMA-init weights. For the original weights, we used cosine learning rate schedule with an initial learning rate of 2c-5 and an end learning rate of 2e-6. For the PMA-init weights, used cosine schedules with initial learning rates of le-5, 2c-5, and 4c-5, all with an end learning rate of 2c-6. We evaluated the trained models using npen-Benchmark. which includes MMLU Henrryyddyykks et al. [202]], LiveBench White et al. [2025] AMC-2023, GPQA Rein
2024]
and LiveCodeBench White et al. [2025). ). as well as our in house evaluation set comprising OOD, Reasoning, and Instruction Following assessments. As shown in Table I with the same learning
Table 1: Comparisons of performance metrics during SFT stage with varying Ir schedules, where cosine scheduler adopted to decay learning rate from peak to Irend (denoted as Irpeak -> Irend). PMA and baseline, stand for whether our PMA-inil technique is employed or not, respectively. IF refers to Instruction Following.
Open-Benchmark
In-house Evaluation
Model
MMLU LiveBench
AMC-2023
GPQA
LiveCodeBench
OOD
Reasoning
IF
Baseline a
86.8
50.5
61.0
39.7
32.6
32.1
36.3
PMA 2-2--
87.1
39.4
34.7
38.8
PMA 4--22-6
87.2
53.2
65.5
54.4
39.7
33.8
33.2
37.3
PMA 6-22.
87.0
51.3
61.4
54.0
39.2
31.8
32.6
37.2
rate, PMA-init significantly outperformed the baseline on both Open- Biee-Bencchark and our in-house evaluations. Notably, on the in-house evaluation set, we observed improvements of over two points in OOD and Instruction Following, and 1.9 1.9-point increase in Reasoning, In the other two experiments
14

---

with different learning rates, we also saw some degree of improvement compared to the baseline, especially with -- which showed gains of 2.7 points on Livebench and 4.5 points on AMC-2023.
However, were unable replicate such significant gains in subsequent experiments with other model sizes, although it did not negatively impact the final downstream model performance. Therefore, low-cost approach, PMA-init worth trying obtain more powerful downstream model.
C Quantitative Analysis on Model Merging
We begin with second-order Taylor expansion of the loss function L(0) around an optimal parameter set f":
T((0= = I(00)= (0 10---)'VV1 -')''"'(0"")) + (0 --0--)=TH(( 0*).
(6)
where H is the Hessian matrix of the loss function evaluated at 0" (the matrix of second partial derivatives), which captures curvature information. Since 0 an optimal point, gradient (0*)) is zero. Thus, the expansion simplifies to:
L( L((") 2 (0--8)TH((08 8*).
Consider & sets of model parameters 0,, 02 ..,..., Ou.
Let the deviation vector of each model from the optimal parameters be 6, 0; - 0* The loss for each model i can then be approximated as:
1 of Hō:
I(O,) I(0') + 2
(8)
The average loss of these k individual models is:
L(O.) L(0)) +
1
Σ
87 Hō.
2k
The parameters of the merged model are Oavg = t/ Σ/= 0g. The deviation of this merged model from the optimal parameters is Davg- lr = 1 E 2i=1 8, The loss for the merged model is approximated by:
T
k
k
((Oang) ~ L(0 +
Σ
8
H
Σ
84
(10)
k
h
i=1
Expanding the quadratic term:
8
H
Hdj
(11)
2
2k
This can be rewritten by separating diagonal and off-diagonal terms:
1
Σ
ST H6:
H8
(12)
21
i=1
For the merged model to have a lower loss than the average loss the individual models, i.c., ((0avg) < _ Ei= L(O,), the following condition must hold:
k
of
H5,
S
of
Ho;
(13)
2
2k 1=1
Multiplying by 2k and rearranging terms, we get:
k
Hб; +
ΣΣ 8
Hd
< k Σ
of
Hd
(14)
1=1
1=1
15

---

Which simplifies to: k
Ho < (k- Σ of Hd.
(15)
Assuming H a positive definite matrix (which is generally true around local minimum), then each term 65 Hb; > 0. The inequality is more easily satisfied if the off-diagona, terms 8 H (for i of j) are predominantly negative. This "negative correlation" in the context of the Hessian means that the deviation vectors point in somewhat opposing directions relative to the curvature of the loss landscape. This mathematical analysis can intuitively interpreted as follows: 1. The effectiveness of model weight merging stems from the fact that different model checkpoints, representing different points in the training trajectory, have explored different local regions or directions within the parameter space. 2. When these explorations exhibit degree of "complementarity" concerning the geometric structure of the loss function (captured by the Hessian and the cross-terms o? Hő,), ;), their average can position the merged model closer to optimal point than the individual models might be average. 3. This helps explain why merging models, particularly those from a stable yet ongoing training phase, often improves performance. The averaging process can smooth out idiosyncrasies of individual checkpoints. This analysis suggests that weight merging is not merely a simple averaging of parameters but rather process that can leverage the geometric structure of the loss landscape and the diversity among the models being merged.
D Limitations
In our study, we thoroughly investigated the potential of model merging in the pre-training phase, offering significant advantages for teams working on large-scale model pre-training to pursue more daring explorations. This is due to the fact that model merging can replicate the benefits of simulated annealing, greatly shortening the exploration period during pre-training. While our experiments were extensive, certain aspects still remain open for deeper research.
In our experiments, we defaulted to using the optimal learning rate derived from the scaling law for model training, without extensively exploring the impact of learning rate on model merging. In our practice, we believe that training with a higher learning rate could lead to a better model through model merging, which aligns with the findings [Sanyal et al. [2024). However, due the high computational cost, we did not further quantify the impact of learning rate on model merging in more detailed manner.
Additionally, this paper primarily focuses on the application of model merging in pre-training. In reality, due innovations RL algorithms (Yu et al. 2025 Yuan et al. 2025 Shao et al. 2024], RL training has become more stable and often involves longer training cycles, during which a series of adjacent weights can be obtained. This paper does not investigate model merging in the context of post-training scenarios, and we leave this aspect for future research.
E Societal Impact
The development of large-scale language models, while technologically promising, carries significant societal implications that necessitate careful consideration. Our research model merging during pre-training offers advancements that not only push the technical frontier but also contribute positively to the responsible development of artificial intelligence.
E.1 Potential Benefits
The primary contribution of our work is a substantial increase in the efficiency of the LLM pre- training process. The Pre-trained Model Averaging (PMA) technique significantly reduces the computational resources and time required to develop high peefforrffrmeece models. This efficiency has several downstream societal benefits:
Accelerating Innovation and Accessibility: By lowering the immense costs traditionally associated with training state-of-the-art models, our methods can help democratize access large-scale Al development. This allows smaller research labs, academic institutions, and companies contribute to the field, fostering broader innovation.
16

---

Enhancing Applications: The accelerated development cycles enabled by our work can lead to faster improvements critical NLP applications. This includes creating more effective and accessible educational tools, developing more nuanced and accurate communication technologies (such as real- time translation), and building more capable assistive technologies for individuals with disabilities.
E.2
Risks and Mitigation Strategies
We also recognize the potential risks associated with large-scale models and believe our work provides avenues for their mitigation.
Computational Energy Costs and Environmental Impact: The training of LLMs is an energy- intensive process with considerable environmental footprint. Our findings directly address this concern. By demonstrating that merged checkpoints can match or exceed the performance of models that have undergone full, lengthy annealing phase, our technique provides method to curtail total training time. Furthermore, the ability of PMA-init to recover from training instabilities prevents the wasteful process of restarting training from scratch. These efficiencies translate directly into reduced energy consumption and smaller carbon footprint for model development.
Algorithmic Bias:
Large language models can perpetuate and amplify existing societal biases present in their training data. While our research does not introduce a new method for bias detection, the efficiency it creates is critical enabler for more rigorous and responsible AI development. By reducing the cost of each training and evaluation cycle, our methods allow development teams to:
Iterate More Frequently: Conduct more frequent and thorough testing for biases within the model.
Dedicate computational and financial resources saved from training towards crucial mitigation tasks, such as curating higher-quality, more diverse datasets and implementing sophisticated fairness-aware fine-tuning techniques.
Reallocate Resources:
In summary, the model merging techniques presented in this paper offer practical path toward more efficient, sustainable, and responsible ecosystem for AI development. By addressing the core challenge computational cost, we empower the research community to not only build more powerful models but also to invest more deeply in making them safe and beneficial for society.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction of our paper accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work, see Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The paper provide the full set of assumptions and complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper fully discloses all the information needed to reproduce the main experimental results, and although the code and data are not provided, it does not affect the main conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Although the code and data are not provided, does not affect the main conclusions of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/ Soaadd/ss//oeeSSmmmssiiiooooee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudddeo Coaee//ddeeeSbmmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars, but experiments were conducted across multiple model architectures and sizes validate the robustness of the method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: primarily used NVIDIA H-series GPUs for training and evaluation. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp////aaarrs...c/a/a//////ttt
Answer: [Yes]
Justification: The research conducted in the paper conforms in every respect to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper does not address societal impact, but it significantly reduces training costs, facilitating the development of better models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: The methods primarily discussed in our paper do not involve data models with a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have appropriately cited the papers relevant to our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre /0tps///e/.oooeeeeeee222//////// for what should or should be described.
24


Reviews:
[{"review_id": "612d4gpJOx", "reviewer": "Reviewer_UT5Y", "summary": "The paper explores several model-merging strategies applied during pre-training of MoE models. The key result is that merging checkpoints while the learning rate is still constant yields final quality comparable to running a full cosine-annealing schedule. Practitioners could skip the annealing tail and save compute. Multiple merging methods achieve similar performance, and the authors identify an “optimal merging window” for each model scale. They also show that merging neither harms continued pre-training nor supervised fine-tuning, and can make SFT more stable.", "questions": "- What is the PMA technique used in Section 4.1? Is it SMA? The information is missing.\n\n- Figure 5 (bottom)—does it show the 1.3 B/13 B run?\n\n- Figure 6. it seems like learning rate is the only important factor for Continue Pretraining. What would happen in the right figure if we use baseline but with 2e-4 to 1e-5? What would happen if one use a more constant learning rate and then perform PMA during CP? Does the final model outperforms the results without PMA (begin with baseline or a PMA init)?\n\n- In Appendix B, the author perform PMA for the model after continue pretraining. How does the baseline perform compared with the PMA after CP?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\n1. A well-resourced lab/institute has open-sourced a comprehensive study, which is valuable to the community.  \n2. Showing that checkpoint merging can replace annealing offers a concrete way to cut pre-training cost.  \n3. Results hold across several model sizes, suggesting the findings generalize.\n\n\n## Weakness\n\n- Important training details are missing: MoE architecture, routing policy, learning-rate schedule, and the composition of the continued-pre-training corpus.  \n\n- How MoE router weights are combined during merging?", "was_revised": true, "final_justification": "The rebuttal addressed most of my concerns. I echo with the reproducibility concern raised by other reviewers. Overall, I find this a good paper that should be accepted.", "rebuttal": "We thank the reviewer for their thorough review and for raising important points regarding the training details of our work. I hope my responses below can address your concerns.\n\n### **Weakness 1: Missing Training Details**\nThe reviewer noted the absence of details on the MoE architecture, routing policy, learning-rate schedule, and the composition of the continued-pre-training corpus.\n\n### **Answer W1:** \n**1. MoE Architecture and Routing Policy**\nWhile we acknowledge that detailed specifications for the Mixture-of-Experts (MoE) architecture and routing policy were not fully disclosed, our conclusions are designed to be **architecture-agnostic**. To ensure robustness, we conducted experiments with both MoE and dense models, where the dense model mirrors the exact architecture of LLaMA 3.1. Our findings hold across both architectures, demonstrating that our contributions are not dependent on specific MoE structural details. Regarding the routing policy, we adopted a standard approach, where *k* out of *N* experts are activated. **No specialized modifications were made**, ensuring our results are generalizable and not tied to a bespoke routing strategy. This strengthens the broader applicability of our findings.\n\n**2. Learning-Rate Schedule**\nThe learning-rate schedule was not detailed in the paper, but we utilized the Warmup-Stable-Decay (WSD) schedule, a widely adopted strategy, applied to our internal dataset. While we regret not including this in the manuscript, our internal experiments across **varied learning-rate schedules consistently validated our conclusions**. This robustness across settings supports our claim that the learning-rate schedule does not materially affect the contributions of this work.\n\n**3. Composition of Continued-Pre-Training Corpus**\nWe acknowledge that the composition of our continued-pre-training corpus was not fully described. However, our experiments were conducted on a diverse internal dataset, and we performed extensive ablations to confirm that **our findings are independent of specific corpus characteristics**. These ablations, while not detailed in the paper due to space constraints, consistently support our conclusions across different data compositions.\n\n### **Weakness 2: How MoE router weights are combined during merging?**\n### **Answer W2:**\nThank you for your question regarding how the router weights are combined during merging. In our work, we **did not perform any specialized merging operations on the router weights**. The merging process was conducted at the model level, consistent with standard practices for both MoE and dense architectures. Our experiments showed no evidence of instability or performance degradation resulting from this approach. This robustness further supports the generalizability of our findings, as the merging process does not rely on bespoke handling of router weights.\n\n### **Question 1: What is the PMA technique used in Section 4.1? Is it SMA? The information is missing.**\n### **Answer Q1:**\nWe apologize for the oversight of not clearly defining this term. To clarify, **PMA refers to Simple Model Averaging (SMA)**, which is the default merging method used in our experiments. We will make corrections in subsequent versions.\n\n### **Question 1: Figure 5 (bottom)—does it show the 1.3 B/13 B run?**\n### **Answer Q2:**\nWe confirm that **Figure 5 (bottom) indeed depicts the results for the 1.3B/13B model run**. We apologize for the lack of clarity in the figure caption and appreciate your feedback. This oversight will be addressed in the revised manuscript by updating the caption to explicitly annotate that the figure corresponds to the 1.3B/13B run.\n\n### **Question 3:Figure 6. it seems like learning rate is the only important factor for Continue Pretraining. What would happen in the right figure if we use baseline but with 2e-4 to 1e-5? What would happen if one use a more constant learning rate and then perform PMA during CP? Does the final model outperforms the results without PMA (begin with baseline or a PMA init)?**\n### **Answer Q3:**\n**1. Effect of Baseline with Learning Rate 2e-4 to 1e-5**\nIn our experiments, the learning rate of 4e-4 was determined to be the optimal hyperparameter through extensive grid search, outperforming other schedules, including 2e-4 to 1e-5. Thus, we opted to **report results with the optimal 4e-4 schedule to best represent the baseline’s potential.**\n \n**2. Constant Learning Rate with PMA During CP**\nRegarding the use of a constant learning rate combined with PMA (Simple Model Averaging) during CP, we can refer to Figure 3, which compares a constant learning rate with PMA against the standard cosine decay schedule. The results indicate that the **constant learning rate with PMA yields performance that is nearly indistinguishable from the cosine schedule, suggesting minimal loss in effectiveness.** We chose not to include this specific combination in Figure 6 to maintain a controlled, single-variable ablation study, isolating the impact of the learning rate schedule while keeping other factors consistent. This design choice ensures clarity in attributing performance changes to specific variables.\n  \n**3. Performance Comparison: Final Model with vs. without PMA**\nOur experiments suggest that, when trained to convergence, **the final model performance is comparable across these settings**. The use of PMA, whether applied at initialization or during CP, does not significantly alter the final performance compared to the baseline, as the models converge to similar optima.\n\n### **Question 4: In Appendix B, the author perform PMA for the model after continue pretraining. How does the baseline perform compared with the PMA after CP?**\n### **Answer Q4:**\nAs shown in Figure 2 (rightmost plot, MoE 15/150B model), **the best-performing PMA model after CP achieves a modest improvement over the best checkpoint of the baseline CP model.** Specifically, the PMA model, which is selected based on the highest evaluation score, outperforms the baseline CP checkpoint, **though the improvement is not substantial.** This suggests that PMA provides a slight enhancement in performance post-CP, likely due to its ability to aggregate complementary strengths from multiple checkpoints. Furthermore, in Appendix B, we observe that when **PMA is used as an initialization for supervised fine-tuning (SFT), the resulting model significantly outperforms the baseline CP checkpoint (non-PMA initialized).**"}, {"review_id": "0GtJoWPNxL", "reviewer": "Reviewer_fgbv", "summary": "In this paper, authors study model merging in LLM pretraining. Useful insights on selecting hyperparameters for model merging, the effects of model merging on pretraining performance and downstream task performance, the benefit of model merging in helping with training stability are provided through extensive experiments.", "questions": "1. Minor: The related work section looks a bit repetitive / is similar in structure with the first few paragraphs of introduction.\n\n2. For figure 2, I did not find the name of the benchmark(s) for which the performance is reported. Perhaps it’s an average performance?\nAuthors frame the experiment section as tackling one research question for each subsection. It would be nice if there are fonts in bold to summarize the findings in each of subsection (currently only 4.1 does that).", "limitations": "n/a", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\nModel merging is under-explored in pretraining stage. This paper provides useful insights and guidelines through extensive experiments to a reasonable scale.\n\n\nWeakness/Questions\n1. The experiment settings are disorganized. Although I can guess the setting from the context for most of times, I encourage authors to systematically include settings for the experiment section. For example, for PMA-init, is it used with warmup-stable phase or annealing phase?\n\n2.\n> pre-training with a constant learning rate, combined with model merging, can200\neffectively match the performance of an annealed model at any point in the training process201\nwithout the need for learning rate annealing.\n\nTo eliminate the need for learning rate annealing as authors claimed, authors should perform experiments on the comparison between (annealed model  + post-training) and (merging constant learning rate model + post-training). If 4.4 is performed using (merging constant learning rate model), then this claim in 4.1 is valid. Otherwise, there should be more experiments to support the claim quoted.\n\n2. On 262~266, I personally have observed similar behaviors on vision classification model pretraining, where EMA improves pretraining metric while the benefit does not transfer to downstream training. Although this result may come as negative, I think it is valuable information to the community and authors can consider discussing more (and include results) on the results from SFT stage. I also encourage authors to conduct more empirical analysis on the reason why PMA’s benefit does not transfer to downstream finetuning.\n\n4. Although model merging between different finetuned models are well-studied, I don’t see works that evaluate EMA/SMA on the same training trajectory during the post-training stage. I wonder how many of these conclusions in pretraining merging transfer to post-training.", "was_revised": true, "final_justification": "I believe the paper will be strengthened by organizing the setting details, which is particularly important for an empirical work like this. I will keep my current positive rating and encourage authors to include these revisions in the camera-ready version.\n\nSome of my questions are listed as future works, which is fine. I find the empirical exploration of this paper valuable to the community.", "rebuttal": "We appreciate the reviewer’s detailed evaluation and valuable feedback on our work’s training details. I hope my responses below adequately address your concerns.\n### **Weakness 1: The experiment settings are disorganized. Although I can guess the setting from the context for most of times, I encourage authors to systematically include settings for the experiment section. For example, for PMA-init, is it used with warmup-stable phase or annealing phase?**\n### **Answer W1:**\nWe fully agree that clearly documenting the settings for the experiment section enhances reproducibility and clarity, and we appreciate your encouragement to improve this aspect of our work. Regarding the specific question about PMA-init, we would like to clarify its usage across different training phases. PMA-init can be applied flexibly depending on the training objective:\n\n  **1. Warmup-Stable Phase:** If the goal is to resume training and enhance model stability, PMA-init can be employed during the warmup-stable phase to initialize the model with robust weights, thereby improving training convergence and stability.\n\n  **2. CPT (Continued Pre-Training) with Annealing Phase:** When used in the context of continued pre-training (CPT), PMA-init can be applied during the stable phase to provide initialized weights for downstream CPT annealing, ensuring a smoother transition to fine-tuning.\n\n  **3. SFT (Supervised Fine-Tuning) Training:** For supervised fine-tuning, PMA-init can be integrated during the CPT annealing phase, where PMA weights are merged to provide a strong starting point for fine-tuning.\n\n  We sincerely appreciate your suggestion and will ensure that the experimental settings, including the specific configurations for PMA-init, are systematically and clearly documented in the revised manuscript. \n\n### **Weakness 2: To eliminate the need for learning rate annealing as authors claimed, authors should perform experiments on the comparison between (annealed model + post-training) and (merging constant learning rate model + post-training). If 4.4 is performed using (merging constant learning rate model), then this claim in 4.1 is valid. Otherwise, there should be more experiments to support the claim quoted.**\n###  **Answer W2:**\nIn Section 4.4, the continued pre-training experiments utilize a merging constant learning rate pre-trained model followed by cosine scheduling for training. For SFT, we employed the merged annealed model with post-training. To further support the claim in Section 4.1, we have conducted additional experiments comparing the (merging constant learning rate model + post-training) setup against the (annealed model + post-training) setup on 3B/30B MoE models. The results are as follows:\n  - MMLU: The merging constant learning rate model + post-training setup achieved a +0.2 point improvement compared to the annealed model + post-training.\n  - LiveBench: A +0.8 point improvement compared to the annealed model + post-training.\n  - GPQA: A slight -0.6 point decrease compared to the annealed model + post-training.\n  These results demonstrate that **the merging constant learning rate model is highly competitive with, and in some cases outperforms, the annealed model, supporting the claim in Section 4.1 that learning rate annealing can be eliminated without compromising performance.** Additionally, as shown in the paper, **the annealed model and the merging constant learning rate model exhibit comparable performance, and the inclusion or exclusion of post-training does not alter the core conclusions of our work.**\n\n### **Weakness 3: On 262~266, I personally have observed similar behaviors on vision classification model pretraining, where EMA improves pretraining metric while the benefit does not transfer to downstream training. Although this result may come as negative, I think it is valuable information to the community and authors can consider discussing more (and include results) on the results from SFT stage. I also encourage authors to conduct more empirical analysis on the reason why PMA’s benefit does not transfer to downstream finetuning.**\n### **Answer W3:**\n  Given the focus of our paper on model merging during pretraining, we prioritized analyzing the impact of PMA in that context due to space constraints and thematic coherence. However, we fully recognize the value of exploring model merging in downstream stages such as SFT and reinforcement learning (RL), and **we are actively preparing experiments to investigate these scenarios further.**\n\n  To address your specific point about the lack of PMA’s benefits in downstream fine-tuning, we hypothesize that this behavior may stem from the typically smaller learning rates used in SFT. In downstream training, where models often follow similar optimization trajectories, merging checkpoints from the same trajectory (e.g., via PMA) may not yield significant differences compared to the original checkpoints. However, we have observed promising results when encouraging diversity in model training. For instance, by training separate models on distinct data sources (e.g., code and math datasets) and performing simple model-level merging, we can achieve a merged model that performs exceptionally well on both tasks—often surpassing the performance of a single model trained on mixed data. **This suggests that model diversity during pretraining or fine-tuning can enhance the effectiveness of merging strategies in downstream tasks.**\n\n### **Weakness 4: Although model merging between different finetuned models are well-studied, I don’t see works that evaluate EMA/SMA on the same training trajectory during the post-training stage. I wonder how many of these conclusions in pretraining merging transfer to post-training.**\n### **Answer W4:**\n  Our paper primarily focuses on model merging strategies, including PMA (a variant of EMA/SMA), during the pretraining phase of large language models, as this stage is critical for establishing robust model weights. This focus was driven by the need to address the unique challenges of pretraining, such as computational efficiency and model stability. However, we recognize the value of exploring these techniques in the post-training stage, including supervised fine-tuning (SFT) and reinforcement learning (RL), as you suggested.\n\n  From a theoretical perspective, we believe that post-training (SFT) is not fundamentally different from pretraining in terms of model merging principles, though it typically involves smaller learning rates and more limited datasets. In post-training scenarios, such as SFT, merging models from the same training trajectory (e.g., via PMA) may lead to faster convergence due to the stabilization effect of averaging weights, similar to what is observed in the continued pre-training stage. However, this may not necessarily translate to better performance, potentially due to the limited diversity in model weights caused by smaller learning rates and constrained data.\n\n### **Question1: Minor: The related work section looks a bit repetitive / is similar in structure with the first few paragraphs of introduction.**\n### **Answer Q1:**\n  We appreciate your observation that the related work section appears repetitive in structure compared to the first few paragraphs of the introduction.\n\n  To address this concern, we will revise the related work section to reduce redundancy and ensure it complements the introduction more effectively. Specifically, we plan to streamline the content by focusing on a more concise and targeted discussion of prior work, emphasizing how it differs from or builds toward our contributions. **We will also restructure the section to avoid overlapping themes with the introduction, ensuring that the related work section provides a distinct and comprehensive review of the literature while clearly positioning our work within the field.**\n\n### **Question 2: For figure 2, I did not find the name of the benchmark(s) for which the performance is reported. Perhaps it’s an average performance? Authors frame the experiment section as tackling one research question for each subsection. It would be nice if there are fonts in bold to summarize the findings in each of subsection (currently only 4.1 does that).**\n### **Answer Q2:**\n We appreciate your observation that the benchmark(s) for the performance reported in Figure 2 are not explicitly named and your suggestion to use bold fonts to summarize the findings in each subsection for clarity.\n\n  To clarify, **the performance reported in Figure 2 represents the average performance across the evaluation datasets described in Section 3 (Experimental Setup).** We apologize for any ambiguity and will update the caption or accompanying text of Figure 2 in the revised manuscript to explicitly state that the results reflect the average performance across these benchmarks, with a reference to Section 3 for details on the specific datasets used.\n\n  Regarding your suggestion to use bold fonts to summarize the findings in each subsection of the experiment section, we agree that this is an excellent way to enhance readability and highlight key results. Currently, only Section 4.1 includes such summaries. In the revised manuscript, **we will incorporate bolded summary statements at the end of each subsection** (e.g., Sections 4.2, 4.3, etc.) to concisely highlight the main findings and align with the research question addressed in each subsection. This will make the experiment section more structured and easier to navigate.\n\n  Thank you again for your thoughtful suggestions, which will significantly improve the clarity and presentation of our work. If you have additional recommendations for refining Figure 2 or the experiment section, we would be happy to consider them in our revisions."}, {"review_id": "5uI9xGMMY3", "reviewer": "Reviewer_DRRf", "summary": "The paper investigates the utility of merging multiple consecutive checkpoints at the pre-training stage of model development. Paper includes experiments with MoEs and dense models (in the appendix). Among other things, authors find that model merging can significantly improve the downstream performance of the models, make CPT more stable, as well make final learning rate annealing unnecessary.", "questions": "- authors claim that PMA is a novel framework (l.71), but wasn't a very similar strategy already been applied in e.g. LAWA? Isn't this just application of the LAWA strategy at scale?\n- Authors say: \"Through extensive experiments, we demonstrate the effectiveness of PMA-init on both CT and SFT stages.\" This sound somewhat misleading to me, i.e. for CT authors show more stable training but no clear performance advantages, for SFT the gains in performance are also very limited and not consistent as acknowledged by the authors;\n- why is the main focus on MoEs and not dense models? Since the paper includes dense mdoels merging in appendix, it would be  nice to include a comparison of the effectiveness of PMA in MoEs vs Dense\n\nFurther limitations & comments:\n- related works is missing an important work [1]\n- authors did not include more sophisticated weight merging methods such as TIES, Fisher merging etc.\n- authors mention significant computation saving of merging but do not quantify the saving \n- Section 4.3 is titled \"How to determine the optimal interval and number of weights to merge for various model\nsizes?\", but the content of the section is not focusing on how to efficiently determine those parameters, but rather on descriptive analysis of the said hyperparameters\n- In the NeurIPS Paper Checklist authors state that experimental results are reproducible, which is not the case given that the model architecture has not been publicly released and is not described in the detail in the paper (ll.130-131).\n\n[1] Yadav, Prateek, et al. \"What Matters for Model Merging at Scale?.\" arXiv preprint arXiv:2410.03617 (2024)", "limitations": "This paper includes experiments with models of very large scale (dense of up to 70-B) but provides no discussion on potential societal impact.\nThe paper does not provide any details on the data that has been used for training the models, no information about copyright, privacy etc. is provided, which is concerning.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 1, "strengths_and_weaknesses": "Overall, the paper makes a valuable contribution by empirically validating the effectiveness of known model merging techniques for pre-training modern LLMs using contemporary pre-training pipelines. I believe the results presented are valuable to the community and practitioners, especially given the substantial compute expenditure on the presented experiments. I found particularly interesting the insight that merging the k latest checkpoints can achieve similar performance gains as complete learning rate annealing. Additionally, the paper is well written, uses simple language and is easy to follow.\n\nThat being said, the main weakness of the paper is its lack of methodological novelty; while empirical validation of these insights is valuable, many insights are somewhat intuitive (merging well converged models is better, more checkpoints etc.).\n\nOverall, while clearly valuable for the practitioners, I am not 100% sure that empirical validation of existing techniques at scale constitute sufficient contribution to NeuIPs.", "was_revised": true, "final_justification": "While I definitely acknowledge the practical relevance of this work, I am concerned by the novelty I acknowledge that scaling existing techniques is challenging in practice) as well as limited reproducibility. Hence I keep my \"weak accept\" assessment.", "rebuttal": "We appreciate the reviewer’s detailed evaluation and valuable feedback on our work. I hope my responses below effectively address your concerns.\n### **Weakness 1 : the main weakness of the paper is its lack of methodological novelty; while empirical validation of these insights is valuable, many insights are somewhat intuitive (merging well converged models is better, more checkpoints etc.). Overall, while clearly valuable for the practitioners, I am not 100% sure that empirical validation of existing techniques at scale constitute sufficient contribution to NeuIPs.**\n### **Answer W1:**\n  We sincerely thank the reviewer for their thoughtful feedback and for recognizing the clear value of our work for practitioners. We address the main concern regarding methodological novelty and the perceived intuitiveness of some insights below, clarifying why our contributions are significant and well-suited for NeurIPS.\n\nWe acknowledge that **our paper prioritizes simplicity over introducing complex, novel techniques. This was an intentional design choice.** By focusing on straightforward model-level merging methods without intricate module-level tricks, we aimed to develop a generalizable approach that is robust across diverse scenarios and model architectures. Complex techniques often risk overfitting to specific settings, limiting their practical applicability. In contrast, our simple yet effective method demonstrates broad utility, which we believe is a key strength. The empirical validation of such simplicity at scale is non-trivial and provides actionable insights for the community, particularly as large-scale models become increasingly prevalent. \n\n  The reviewer notes that some insights, such as \"merging more checkpoints is better,\" may seem intuitive. While this observation may appear straightforward in hindsight, our experiments reveal important nuances. **For instance, merging more checkpoints often implies larger temporal gaps between checkpoints, which can introduce unforeseen challenges, such as compatibility issues or degraded performance in certain tasks.** Our work rigorously quantifies these trade-offs, providing empirical evidence to validate or challenge intuition. As reviewer says, they offer actionable guidance for practitioners. \n\n  Moreover, **scaling insights from smaller models to larger ones is far from trivial.** Our work validates these insights at scale, showing that principles effective for smaller models hold for larger, more complex models and datasets. This scalability is not guaranteed, and our comprehensive experiments provide valuable evidence of robustness, making our results highly insightful for the field. **Meanwhile, we have also done extensive explorations in areas such as applying model merging to downstream training and addressing stability issues. To the best of my knowledge, no previous studies have conducted such explorations.**\n\n  We believe our contribution aligns with NeurIPS’ mission to advance AI by disseminating impactful, practical knowledge. While methodological novelty is valuable, empirical validation of scalable, generalizable techniques is equally critical for driving real-world progress in large language model development. **Our work delivers actionable, reproducible insights that benefit the broader AI community.** We hope NeurIPS will serve as a platform to amplify these findings, fostering further innovation and adoption in the field.\n\n### **Question1: authors claim that PMA is a novel framework (l.71), but wasn't a very similar strategy already been applied in e.g. LAWA? Isn't this just application of the LAWA strategy at scale?**\n### **Answer Q1:**\nWe acknowledge that our description on line 71 may have caused some ambiguity. To clarify, our intent was not to claim that PMA introduces entirely new methodological concepts but rather to emphasize its novel application and design as a streamlined, scalable framework for model merging. LAWA demonstrated that merging checkpoints at intermediate stages can significantly accelerate training. In this paper, **we not only explore scaling further, which is a non-trivial endeavor, but also conduct experiments on additional merging details, such as the interval between merges. Additionally, we perform extensive explorations in areas such as applying model merging to downstream training and addressing stability issues. Overall, we believe our paper differs significantly from LAWA in multiple aspects.**\n\n### **Question2 : \"Through extensive experiments, we demonstrate the effectiveness of PMA-init on both CT and SFT stages.\" This sound somewhat misleading to me, i.e. for CT authors show more stable training but no clear performance advantages, for SFT the gains in performance are also very limited and not consistent as acknowledged by the authors;**\n### **Answer Q2:**\n  We acknowledge that the phrasing could be clearer and offer the following clarifications:\n -  PMA-init achieves significant improvements of ~2 points on challenging OOD and Reasoning tasks in SFT, which are **substantial given the difficulty of these benchmarks.** While gains vary across tasks, these results highlight PMA-init’s effectiveness.\n - Generally, **PMA-init enhances training stability without performance loss, enabling more reliable training and occasionally yielding stronger models, demonstrating its practical value.**\n### **Question3: Why is the main focus on MoEs and not dense models? it would be nice to include a comparison of the effectiveness of PMA in MoEs vs Dense**\n### **Answer Q3:**\n  The primary focus on MoE architectures in our paper is motivated by their increasing prominence in recent advancements in large-scale language models. Over the past six months, **MoE has emerged as the dominant architecture for flagship models due to its scalability, computational efficiency, and strong performance in diverse tasks (e.g., like Deepseek-V3, Qwen3 and others).** Given this trend, we believe that studying PMA in the context of MoEs aligns with the future direction of large model development, making it a timely and impactful contribution.\n\n  That said, we fully recognize the importance of dense models, which remain widely used and relevant. To ensure a comprehensive evaluation, we included experiments on dense model merging in Appendix A. **These results demonstrate that PMA achieves consistent effectiveness across both MoE and dense architectures, highlighting its versatility.**\n\n### **Question 4:related works is missing  [1]**\n### **Answer Q4:**\n  Thank you for pointing out the omission of the important work [1] in our related works section. We sincerely apologize for this oversight.  In the revised version of the manuscript, we will incorporate a discussion of [1] in the related works section, highlighting its contributions and relevance to our research. \n[1] Yadav, Prateek, et al. \"What Matters for Model Merging at Scale?.\" arXiv preprint arXiv:2410.03617 (2024)\n\n### **Question5: did not include more sophisticated weight merging methods**\n### **Answer Q5:**\n  We acknowledge that our paper prioritizes simplicity over introducing complex, novel techniques. Refer to **Answer W1**.\n\n### **Question 6:authors mention significant computation saving of merging but do not quantify the saving**\n### **Answer Q6:**\nIn the manuscript, we reference Figure 2, which illustrates that during the CT phase, our proposed PMA approach achieves performance comparable to full annealing while training on only 50% of the token volume. **This effectively translates to a 50% reduction in computational cost for the CT phase compared to the baseline full annealing method.** We believe this constitutes a significant saving in computational resources.\n\n### **Question7: Section 4.3 is not focusing on how to efficiently determine those parameters, but rather on descriptive analysis of the said hyperparameters**\n### **Answer Q7:**\n In practical model training, it is challenging to precisely determine the optimal merging hyperparameters for specific model sizes. This is because, in large-scale model training, it is infeasible to densely save checkpoints for merging due to computational and storage constraints. Therefore, we can only provide qualitative results. **Generally, larger models require larger merging intervals, and merging around 10 weights is typically sufficient. We believe this level of insight is adequate to provide developers with a solid understanding.**\n\n### **Question 8:In the NeurIPS Paper Checklist authors state that experimental results are reproducible, which is not the case given that the model architecture has not been publicly released and is not described in the detail in the paper (ll.130-131).**\n### **Answer Q8:**\n  While we acknowledge that these details were not fully disclosed, our conclusions are designed to be architecture-agnostic. To ensure robustness, we conducted experiments with both MoE and dense models, where the dense model mirrors the exact architecture of LLaMA 3.1. **Our findings hold across both architectures, demonstrating that our contributions are not dependent on specific MoE structural details.**\n\n### **Question9: This paper includes experiments with models of very large scale (dense of up to 70-B) but provides no discussion on potential societal impact. The paper does not provide any details on the data that has been used for training the models, no information about copyright, privacy etc. is provided, which is concerning.**\n### **Answer Q9:**\n We fully agree with the reviewer on the importance of transparency and discussing societal impacts, and we appreciate your pointing out these areas for improvement. Our primary intention is to share technical details to contribute to the academic community, rather than directly promoting specific application scenarios or commercial deployment. **Therefore, we believe the conclusions of the paper are independent of specific data and do not pose significant societal risks.**"}]

Author rebuttal:

Meta-review: This paper presents a comprehensive empirical study of model merging in the pre-training of large language models, covering both dense and mixture-of-experts architectures up to 100B+ parameters.

In this work, the authors show that merging checkpoints during the constant learning rate phase can effectively replace full cosine annealing schedules, reducing training cost while maintaining or improving final model quality. Strengths include the scale and breadth of experiments, practical insights for practitioners, and the demonstration that merging improves training stability and does not harm continued pre-training or fine-tuning.

Weaknesses concern limited methodological novelty and missing details on architectures and datasets that limit reproducibility. During rebuttal, the authors added clarifications, new baselines, and expanded analysis of learning-rate effects, addressing most reviewers’ concerns.

Overall, the paper offers valuable empirical guidance on the model merging technique for efficient large-scale model training and would be of interest to the LLM community, and I recommend acceptance.
Decision: Accept (poster)