Maximizing Intermediate Checkpoint Value in LLM Pretraining with Bayesian Optimization
Authors: Deyuan Liu
Zecheng Wang
Bingning Wang
Weipeng Chen
Chunshan Li
Zhiying Tu
Dianhui Chu
Dianbo Sui
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models (LLMs)
Pretraining
Checkpoint Merging
Bayesian Optimization
Primary area: deep_learning->large_language_models

Abstract: The rapid proliferation of large language models (LLMs), such as GPT-4 and Gemini, underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environmental costs. In this paper, we introduce a novel checkpoint merging strategy aimed at making efficient use of intermediate checkpoints during LLM pretraining. This method utilizes intermediate checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various experiments, we demonstrate that: (1) Our proposed methodology exhibits the capacity to augment pretraining, presenting an opportunity akin to obtaining substantial benefits at minimal cost; (2) Our proposed methodology, despite requiring a given held-out dataset, still demonstrates robust generalization capabilities across diverse domains, a pivotal aspect in pretraining.

Maximizing Intermediate Checkpoint Value in LLM Pretraining with Bayesian Optimization
Deyuan Liu *1 Zecheng Wang?*
Bingning Wang? Weipeng Chen
Chunshan Li' Zhiying Tu Dianhui Chu
Dianbo Sui (
deyuanliu @tt...i..dducn
'rechengwang@stti wtt..i..esstuu....uu...
Abstract
56.02
56.05
The rapid proliferation of large language models (LLMs), such GPT-4 and Gemini, underscores the intense demand for resources during their training processes, posing significant challenges due to substantial computational and environ- mental costs. this paper, we introduce a novel checkpoint merging strategy aimed making efficient use of intermediate checkpoints during LLM pretraining. This method utilizes intermediate checkpoints with shared training trajectories, and is rooted in an extensive search space exploration for the best merging weight via Bayesian optimization. Through various exper- iments, demonstrate that: (1) Our proposed methodology exhibits the capacity augment pretraining, presenting opportunity akin to obtaining substantial benefits at minimal cost; (2) Our proposed methodology, despite requiring a given held-out dataset, still demonstrates robust generalization capabilities across diverse domains, a pivotal aspect in pretraining.
$3.06
$4.4
505422
22.65
Accurat
50.02
00.9
40.47
51.33
26.40
46:15
25
Vanilla
Merged
CRpt,-2
Ckpt,
24208
Loss Landscape
Training Step
Figure 1: Overview of the Bayesian optimization framework for checkpoint merging in LLM prefraining, The framework operates by linearly combining intermediate checkpoints i and O, with optimized merging weights A, Through iterative Bayesian optimization, the method identifies performance "sweet spots" in the loss landscape that enhance model efficacy without much additional computational resources, effectively transforming intermediate checkpoints into improved models.
instance, training the LLaMA2 70B model with 2T tokens necessitates 1,720,320 GPU hours (Touvron et al., 2023), and the development of a transformer with 213 million parameters through neural architecture search can lead to environmental burdens equivalent to the lifetime CO2 emis- sions of five cars over their entire lifespans (Strubell et al., 2019; Faiz et al., 2023). Consequently, making efficient use of checkpoints and the intermediate stages of the pretraining process has emerged as key challenge in this field.
1. Introduction
With the rapid development of L.Mss, such GPT-3 (Ope- nAI, 2023), GPT-4 (OpenAI et al., 2023), PaLM (Chowd- hery et al., 2023) and Gemini (Team et al., 2023), which boasts tens to hundreds of billions of parameters, the demand for new LLMs and the research aimed at enhancing their capabilities have significantly increased. But we should note that the training requirements for these LLMs are substantial, not only in terms of computational resources, human resources, and capital resources, but also regarding energy consumption and environmental impact. For
In response to this challenge, researchers have adopted various strategies in LLM pretraining, including mixed- precision training (Shocybi et al., 2019), zero-redundancy optimizer (Rajbhandari et al., 2020), continuous retrain- ing (Qin al., 2022), pipeline parallelism (Liu et al., 2023). and depth up-scaling methods (Kim al., 2023). Although these approaches contribute to efficient pretraining by optimizing model architecture and processes, they primarily focus on structural or optimization improvements rather than directly enhancing the utilization of intermediate checkpoints in the pretraining phase (Hou et al., 2022).
"Equal contribution 'arbbin Institute of Technology "Tencent Wechat IIneppeddent Researcher. Correspondence to: Dianbo Sui @ hit.edu.cn?.
Unlike these studies, we focus on the model merging strat- egy, a classic topic in machine learning (Utans, 1996; Chen et al., 2017; Wortsman et al., 2022; Akiba et al., 2024; Li
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Maximizing Checkpoint Value with Bayesian Optimization
132200 33000 37000 5900 22000 /42200
pretraining by efficiently utilizing intermediate check- points;
35
Score Comparison Perspective
34.95
181.76
(b) Besides superior performance, the merged soup deter- mined by specific held-out dataset same Wortsman et al. (2022); Matena & Raffel (2022), still exhibits strong generalization capabilities across various do- mains, a crucial aspect in pretraining.
50 62
56.65
56 68
02
38.29
50.92
001
113 13.000
(1: 700
(13443)
Contributions: summary, the contribution of this paper is threefold:
53.05
53.00
53.62
53:15
53:31
11000
32.39
6-2
(3.99)
(2.45
(:2.64)
(a) We propose merging checkpoints in the pretraining trajectory to make efficient use of checkpoints, offering substantial improvements without additional resource requirements.
8808
46.35
1e1.221
50.00
10.76
91 11.29
51 09
(2.85)
51.50
14.33
50.03
(4.15)
14.05)
90-17
2208 4408 6605
10.47
48.16
03:28
40.80
43.70
68:30
17:37
05.53
34.32
(19.22)121.000
24.34)
30.113
(b) To find the optimal merging weight, we leverage Bayesian optimization, which excels optimizing ex- pensive black-box derivative-free objective functions.
23.09
67.722
25.40
25.29
25.23
23.26
25.26
25.28
25.26
14])-30
29.09
525
25.25
72/033000
24.77
00.
25
2200
4408
6608
8800
11008
13200 15400
17608
19800
22008 24208
(c) Through various experiments, we denote our method exhibits superior performance and the newly merged checkpoint maintains strong generalization across dif- ferent domains.
Figure 2: Performance landscape of pairwise checkpoint merg- ing using the Greedy Soup method on the C-Eval benchmark across 11 Baichuan2 checkpoints spanning 200B to 2640B to- kens. The heatmap reveals that merging adjacent checkpoints (near the diagonal) generally yields superior performance, while merging distant checkpoints results in significant performance degradation.
2. Pilot Experiments
We conducted comprehensive pilot experiments to address two fundamental research questions that guide our check- point merging strategy:
et al., 2023b; Yang et al., 2023b), to enhance LLM pretrain- ing in this paper. In particular, we employ checkpoints saved during pretraining and average these checkpoint parame- ters to improve pretraining without requiring substantial re- sources, since merged checkpoints can reduce the variance of the combined output relative to the output of the individ- ual checkpoint while not increasing the bias (Utans, 1996).
RQ: Which checkpoints along the pretraining tra- jectory should be merged? RQ2: How should these checkpoints be merged opti- mally?
However, conducting checkpoint merging is not trivial in pretraining, because different local minima may be found in averaging parameters (Utans, 1996; Chen et al., 2017). Therefore it is important to investigate the basic characters of checkpoint merging and wisely determine the merging weight.
2.1. Experimental Setup
To systematically investigate these research questions, we se- lected eleven representative checkpoints from the Baichuan2 model ( (Yang et al., 2023a), spanning a comprehensive range from 200B to 2640B tokens during pretraining. We eval- uated the merged checkpoints using C-Eval (Huang et al., 2023), rigorous benchmark encompassing 52 subjects across four difficulty levels, providing comprehensive cov- erage of language understanding capabilities. All merging experiments employed the greedy soup strategy (Wortsman et al., 2022), where checkpoints are combined sequentially, with each checkpoint added only if it demonstrates measur- able improvement in accuracy on a held-out development set. 2.2. IQQ: Strategic Identification of Mergeable Checkpoints
Our Approach. To this end, we make the following effort: (1) we conduct some pilot experiments to explore the characters of checkpoint merging; (2) Based on the findings in the pilot experiments, we propose method rooted in Bayesian optimization to find the optimal or near-optimal merging weight. In detail, we first explore two research questions: Which checkpoints in the pretraining trajectory should be merged? and How to merge checkpoint? via various pilot experiments. Then, based on findings in pilot experiments, we leverage Bayesian optimization to optimize the expensive, black-box, and derivative-fres objective function of checkpoint merging, and determine the checkpoint merging weight.
To systematically address RQQ,, we conducted exhaus- tive exploration of all pairwise combinations among the
Through various experiments, we mainly find that:
According Wortsman al. (2022), the merged result called "soup".
(a) Our proposed approach has the potential to enhance
2

---

Maximizing Checkpoint Value with Bayesian Optimization
2.3. IRQ2: Optimal Strategies for Merging Checkpoints
54.98
54.82
To explore IQ2, we examined the impact of varying merg- ing weights when combining two representative pairs of checkpoints: Baichuan2- Saraaa22-554000 1540B with Baichuan2-1760f and Baichuan2-2200B with Baichuan2-2420B, For each pair, we uniformly sampled 100 weights from the interval [0, 1] and evaluated each merged model on C-Eval, illustrated in Figure 7.
55
Accuracy
54
54.40
53
Results for Distant Checkpoints. When merging check- points with a substantial performance gap, such as Baichuan2- 1540B and Baichuan2- 1720,, the accuracy in- creased smoothly as the weight on the stronger checkpoint (Baichuan2- 1760B)))7BBB increased. Notably, 13 % of the tested weights resulted in improvements beyond the stronger base model.
62
50.92
Merging Baichuan2 Checkpoints 17806 with 1540B Merging Baichward Checkpoints 24208 22008
50
Results for Similar Checkpoints. In contrast, when merg- ing two similarly strong checkpoints (Baichuan2-22008 and Baichuan2-2420B) the relationship between merging weight and performance did not follow striet monotonic pattern. Instead, a broad range of weights 76% led to performance improvements over the stronger checkpoint. A similar pattern was observed in experiments merging DeepSeek's 7B checkpoints (DeepSeck AI et al., 2024) at 180B and 2000B tokens (see Appendix F).
0.0
0.2
0.6 Merging Weight
0.4
0.8
1.0
Figure 3: Impact of varying merging weights on model perfor- mance when combining two representative checkpoint pairs: maichuan2-1540F with Baichuan2-17600 and 2-2000 with Baichuan2-24222 2420B. The graph illustrates accuracy on the C- Eval dataset as a function of uniformly sampled merging weights ranging from O to 1. The results demonstrate distinct patterns: for checkpoints with performance gaps, optimal weights favor the stronger model, while for similarly performing checkpoints, broad range of weights (76%) ) can yield improvements, highlight- ing the complexity of the optimization landscape.
Summary Our pilot experiments provide valuable in- sights into the strategies for merging model checkpoints. Specifically, merging adjacent checkpoints along the pre- training trajectory generally enhances performance, while merging distant checkpoints can be detrimental. Addition- ally, the choice of merging weights plays a crucial role, especially when combining checkpoints of similar strength, where a wide range of weights can yield performance gains. These findings inform our approach to checkpoint merging in subsequent methods.
eleven Baichuan2 checkpoints, yielding 55 distinct merg- ing scenarios. The performance of these merged models was rigorously evaluated on the C-Eval test set, with results visualized in Figure 2.
Key Findings. Our analysis reveals that merges involy- ing adjacent checkpoints in the pretraining trajectory consistently led to substantial performance improvements compared to individual models. Most notably, merging Baichuan2- 1980B with 2aichuan2-2200B achieved an ac- curacy of 56.65% %. significantly outperforming Baichuan2- 2200B alone (54.98%) %). Remarkably, this merged model surpassed even the final checkpoint, Baichuan2-24203 (54.82%), )), by a substantial margin of 1.83%. These encour- aging trends were consistently observed across the CMMLU benchmark (detailed analysis in Appendix J).
Additional Analyses Further analyses the CMMLU benchmark and DeepSeek's 's checkpoints are provided in Appendix J and Appendix F, respectively, demonstrating the generalizability of our findings across different evalua- tion metrics and model configurations.
3. A Bayesian Approach to Checkpoint Merging in LLM Pretraining
In this section, we present our novel framework for check- point merging during the pretraining of large language mod- els (LLMs). We begin by formally defining the checkpoint merging process. Next, we introduce our optimized ap- proach that leverages Bayesian optimization to determine the optimal merging weights. Finally, we explore the advan- tages of our method in generalization, with detailed proofs provided in Appendix A.
Conversely, merging distant checkpoints often resulted in severe performance degradation. For instance, com- bining the early-stage Baichuan2-220B with the mature Baichuan2-2200B yielded an accuracy of merely 25.26%, only marginally exceeding the undertrained Baichuan2- 220B baseline (23.89%) %), indicating destructive interference between disparate training states.
3

---

Maximizing Checkpoint Value with Bayesian Optimization
Algorithm 1 Checkpoint Merging via Bayesian Optimization
Input: Initial checkpoints 1 O, validation dataset D. search bounds (a, 1]. number of iterations N
2: Evaluate initial merging weights X/ (e.g., Xe X 1) and collect observations O = ={(2)) f(( M
3: for k == Ko + to N do
4: Fit a Gaussian Process (GP) to the current observations
5: Select the next merging weight X
arg maxx. A(Ar) using the acquisition function A
6: Merge checkpoints: (.²²) = 2 (*) θr + (1- X 00-1
7: Evaluate the performance f(x, (k)) of 6,
the validation dataset DD
8: Update the observations: O = oU [(2
(*) ((
9: end for
10: Output: Optimal merging weight X; = arg m (f(2)) I (Ae. E0}
3.1. Checkpoint Merging
Empirical evidence indicates that the merged checkpoint + can outperform the most recent checkpoint O when op- timal or near-optimal X is selected. The primary challenge lies in accurately determining this optimal merging weight X which we address by leveraging Bayesian optimization to maximize the effective utilization of intermediate check- points in the subsequent subsection.
During the pretraining of LLMs, the model periodically saves checkpoints denoted as {01, .... ). represent- ing the model parameters at various training iterations. A straightforward strategy to enhance model performance in- volves linearly combining these checkpoints in the parame- ter space, a technique commonly referred to as Checkpoint Soup. This can be mathematically expressed as:
Furthermore, our theoretical analysis ( Appendix A pro- vides insights into why checkpoint merging enhance model performance and convergence. Under realistic as- sumptions about the neural network loss landscape (such as smoothness, quadratic approximation near local minima, and bounded Hessians), we derive tighter bounds on the performance of the merged model θ,. Specifically, we establish that:
,, = E O,, s.t.
===1
1=1
where A are the merging weights assigned to each check- point.
However, the number of checkpoints increases, efficiently utilizing them becomes increasingly challenging due to the nigh-dimensional optimization problem that emerges from the growing number of weighting coefficients (A:). In the general scenario, merging t checkpoints necessitates the optimization of weights A2......... } subject to the constraint Σi= A₂ 1. This constitutes a (t 1)- dimensional optimization problem, which rapidly escalates in complexity as & grows. The exponential expansion of the search space renders exhaustive search and grid search methods computationally impractical for large t. And we empirically validate the efficacy of the pairwise merging strategy, which show Appendix 5.3.
r( R (.. -1- (0) (3)
where (()) denotes the expected performance metric (e.g.. accuracy), and A, captures higher-order terms related to the curvature of the loss landscape and the distance between checkpoints.
Besides, our convergence analysis ( (Appendix B demon- strates that checkpoint merging can influence the conver- gence behavior of gradient- -bsed optimization algorithms. By interpolating between consecutive checkpoints, merging can effectively perform larger steps in parameter space, po- tentially accelerating convergence under certain conditions. Additionally, merging can smooth out fluctuations in the op- timization trajectory caused by stochastic gradients, leading to more stable convergence.
To address this, we adopt a pairwise merging strategy, which significantly simplifies the optimization landscape by stricting the merging process to only the two most recent checkpoints at each step. This approach effectively reduces the problem to a one-dimensional search over the merg- ing weight dy. thereby enhancing computational efficiency and scalability. Formally, the pairwise merging process is defined as:
3.2. Bayesian Optimization for Determining Merging Weights
Accurately determining the optimal merging weight X that maximizes the model's performance is the central challenge in checkpoint merging. To address this, we employ Bayesian optimization, an effective global optimization strategy for functions that are expensive to evaluate or lack closed-form expressions (Frazier, 2018).
+ = x,O, + (1 0.)
where d₂ E (a, 1] and a E (0, 1) serves as a lower bound to constrain the search space.

---

Maximizing Checkpoint Value with Bayesian Optimization
Optimization Objective problem as:
We formulate the optimization
of Improvement (PI), and Upper Confidence Bound (UCB). These are formally defined as:
arg max f ((o( A.E(a.1
(4)
Es [max( f2: [2,0] P(f(A)) > f.)) (e( 3.
for El for PI for UCB
A((A))
(8)
where S( (Xe) represents the performance of the merged model on a validation dataset as a function of the merg- ing weight X, The performance metric could be accuracy, perplexity, or any relevant evaluation metric for LLMs.
where AA()e) represents the acquisition function used to select the next weight X ( f is the best observed per- formance up to iteration k; pa and σx denote the mean and standard deviation of the posterior distribution at A respectively, and / is a tunable parameter controlling the apploration-epplotaaaee
Search Space Constraint By setting A E [a, 1). we constrain the search space to merging weights that retain significant contribution from the latest checkpoint θ, This aligns with practical observations that heavily weighting e often leads to better performance, as it captures the most recent training progress.
trade-off.
To dynamically select the most promising acquisition func- tion, we employ the GP-Hedge strategy, which combines multiple acquisition functions based on their past perfor- mance. At each iteration k, GP-Hedge updates the cumula- tive reward R2 for each acquisition function i:
Gaussian Process Regression We model the objective function f()+) using Gaussian process (GP) regression (Ras- mussen, 2003), which provides a probabilistic framework for modeling unknown functions. The GP defines a prior over functions, characterized by mean function no ()() and covariance function (.) We assume:
M
exp(n/R(())
AGP-Hedge (At)
Σ M Σド=1 (x)(nR,(k))
A.2A)
(9)
where M is the number of acquisition functions (here, M 3), A,()e) represents the individual acquisition func- tions (EI, PI, or UCB), R, (k(k) is the cumulative reward for acquisition function at iteration k, and controls the learn- ing rate. This strategy allows the acquisition function to adapt dynamically, leveraging the strengths each compo- nent based on their historical performance.
(5) where (aa) is the prior mean (often set to zero), and k(As, X)) is the kernel function encoding assumptions about the smoothness of (2x).
[(Ne)~ ~ GP (po(Ar), X())
We collect a set of observations Dots = (( H by evaluating the performance at N different merging weights. The GP posterior mean y (Ae) and variance ox (A+) are updated based on these observations:
The overall procedure is summarized in Algorithm 1. We begin with initial observations (e.g., evaluating X a and X 1) and iteratively select new merging weights using Bayesian optimization until convergence or predefined budget of evaluations.
(Ne) mo(Ae) KT (A))K-((E (f- Mo), ()x) = k(A,, T T (AOK-
(6)
(7)
3.3. Impact on Model Generalization
where k(Ac)= = [K(AA,, )("",..., K(A,,,,,,,,........ IT K is the covariance matrix the observations with elements K,,
Checkpoint merging aims not only to improve performance the validation set but also to enhance generalization to unseen data. Our analysis Appendix D) employs the PAC- Bayesian framework to derive generalization bound for the merged model θ,,
k(xs") xx()). and f, Mo vectors of observed performance values and prior means, respectively.
Our theoretical analysis ( Appendix C) demonstrates that using Ga-based Bayesian optimization can efficiently con- verge to the optimal merging weight Aj. By capturing the function's 's behavior, including any non-convexities or multi- modalities, GPs enable the discovery of A values that yield better performance than those found via grid random search.
PAC- Generalization Bound We define the prior distribution I' and posterior distribution Q over the model parameters as follows:
P= N Q - o??1)
(10)
Acquisition Function Selection The acquisition function determines the next merging weight A to evaluate by bal- ancing exploration of the search space and exploitation of known nigh-performing regions. We consider three acqui- sition functions: Expected Improvement (EI), Probability
where where,V(p, Σ) denotes a Gaussian distribution, and I is the identity matrix. The KL divergence between Q and P is:
X² 10: - 1
DkI(Q11 p)
(11)
20
5

---

Maximizing Checkpoint Value with Bayesian Optimization
Table 1: Comprehensive performance comparison of different checkpoint merging methods applied Baichuan2 and DeepSeek models across multiple benchmark datasets. The table evaluates four checkpoint pairs using various merging strategies: Uniform Soup. Greedy Soup. Fisher Weighted Averaging, RegMean, and our proposed Bayesian Optimization-ban based method. Underlined scores indicate the highest performance achieved by baseline methods within each model pair. Red highlighted improvements demonstrate our method's s consistent superiority, achieving average improvements across different model pairs and maintaining robust performance across diverse evaluation metrics.
Dataset
Baichuan2219900
Raichuan2-22001f Uniform Soup
Greedy Soup
Fisher
RegMean
Ours
C. Eval(S -hoot)
55.63
54,98
$3,00
55.63
55.73
55.21
56.17(+00.44)
(MMLLU(S-hhol)
55.68
56.29
54,20
56.13
55,21
+0.8%(+
MMLL(S-shot)
54,00
51.27
54.30
55.39
54.25
54.77
55.44(+0.05)
GSMSK(4 shot)
23.28
21.99
23.96
23.28
20.92
23.73
24.02(+0.29)
Average
47.15
46.13
46.37
47.65
46.76
47.23
48.13(+0.48)
Dataset
Baichuan2-22008
Baichuan2- Uniform Soup
Greedy Soup
Fisher
RegMean
Ours
C-Fval(5-shat)
54.98
54.82
54,93
55.64
54.44
54.55
56.23(+0)559)
CMMLU(5-shor)
56.29
56.78
56.71
56.78
56.62
56.46
56.97(+0.11)
MMLU(--Shot)
51.27
53.97
54.62
54.82
54.16
54.77
54.56(-0.26)
GSM8K(4-shoo)
19.64
21.00
20.92
21.92
22.44
23.88
(4.32(+1)444)
Average
45.55
46.64
46.
47.29
46.92
47.42
48.02(+0.50)
Dataset
DeepScck-1400t
DeepSeck-16000
Uniform Soup
Greedy Soup
Fisher
RegMean
Ours
C-Eval(5-shot)
38.80
39.40
41.26
40.70
40.24
39,55
+0.75(+0..55)
CMMLU(5 -hotl)
40.27
40.94
42.18
42.25
41.76
41.80
42.55(+00.30)
(ML.U((-shot)
41,94
43.87
43.88
43,95
43.27
43.85(00.03)
(SMSK(4-shot)
11.30
13.27
14.18
14.03
15.39
15.04
15.70(+0.41)
Average
33.08
34.05
35.37
35.22
35.34
34.92
(5.97(+
Dataset
DeepSeek-18005
DeepSeck-20008
Uniform Soup
Greedy Soup
Fisher
RegMean
Ours
C-Eval(5 shot)
43.05
44.36
44.61
44,70
44.81
43.95
45.82(+1101)
CMMLU(S-shot)
45.31
46,82
46.84
46.82
46.49
47.12
47.15(+00.0))
47.68
49.29
49.02
49.29
48.73
49.07
49.43(++011)
GSM8K(4-shot)
16.60
18.88
17.82
18.88
18.73
18.56
19.04(+0.22)
Average
38.16
39.84
39.57
39.92
39.69
39.68
40.36(-0.44)
Using the PAC-Bayesian generalization bound (McAllester, 1998), it holds with probability at least 1 1 8:
MMLU (Hendrycks et al., 2020), and GSM8K (Cobbe et al., 2021), PIQA (Bisk et al., 2020), WinoGrand (Sak- aguchi et al., 2021), SciQ (Welbl et al., 2017), and ARC- Easy (Clark et al., 2018).
DKL (Q (1 P)+ + In
Ee-q [CD(O)] < Ls(Q)
Baseline Merging Methods. We compare against strong baseline merging methods, including Uniform Soup and Greedy Soup (Wortsman et al., 2022), Fisher Weighted Averaging (Matena & Raffel, 2022), and RegMean (Jin et al., 2022). Uniform Soup evenly averages model parameters. Greedy Soup incrementally adds checkpoints that improve performance on held-out set.
2n
(12)
where (+) the expected loss on the data distribution, (s) the empirical loss on the training set, and is the number of training samples.
Since A < 1, the KL divergence in (Eq. (11)) is reduced compared to using O alone. This leads to tighter gen- cralization bound in (Eq. (12)), indicating that checkpoint merging can improve generalization by effectively regular- izing the model.
4.1. Main Results
Table presents the comprehensive results of our merg- ing experiments, highlighting two strategically selected checkpoint combinations that yield substantial performance enhancements. By merging the Baichuan2-19800 and Baichuan2-22000 checkpoints, our method achieves re- markable 0.59 % improvement on the CMMLU dataset com- pared to the Baichuan2-220099 checkpoint alone. Simi- larly, merging the Baichuan2 2-200B and Baichuan2-24200 checkpoints results in a significant 0.59% improvement on the C-Eval dataset compared the Baichuan2-24200 checkpoint. These results underscore the efficacy of our approach in enhancing model performance during the mid to late stages of pretraining.
4. Experiments
Setup
Datasets & LLM Checkpoints. We evaluate our method using multiple pretraining checkpoints and a broad range of datasets. For models, we follow previous Baichuan2 (Yang et al., 2023a) 7B, DeepSeck (DeepSeek-AI et al.. 2024) 7B and Pythia (Biderman et al., 2023), ranging from 70M to 6.9B parameters models. For benchmarks, we evaluate on C-Eval (Huang et al., 2023), CMMLU (Li et al., 2023a),
6

---

Maximizing Checkpoint Value with Bayesian Optimization
Comparison with Baseline Methods. Beyond surpass- ing individual baseline checkpoints, our method signifi- cantly outperforms existing merging baselines across di- verse datasets. Specifically, on the CMMLU dataset, merg- ing the 1aich and Baichuan2-22008 check- points using our approach leads to improvements of 2.68 %, 0.59%, 0.75% and 1.67 % over Uniform Soup, Greedy Soup. Fisher Weighted Averaging, and RegMean, respec- tively. A consistent trend is observed on the C-Eval dataset, where our method outperforms the baselines by substantial margins, demonstrating superior effectiveness in checkpoint merging optimization.
Table Out-of-domain generalization analysis of merged checkpoint soups across language and task boundaries. Merg- ing weights are optimized using the C-Eval dataset (Chinese, in- domain), then evaluated MMI.U, and GSM8K. The IND/OOD format shows in-domain ver- sus out-of of-dnmi performance. The A metric quantifies the total absolute performance difference across domains, where lower val- ues indicate superior generalization stability. Our method achieves the smallest A (0.87), demonstrating robust cross-lingual and cross-task generalization capabilities compared to bascline ap- proaches.
out-of-domain datasets: CMMLU,
Greedy Soup (IND/OOD)
Fisher (IND/OOD)
Ours (IND/OOD)
Dataset
Cross-Archteccture Generalizability. To validate the ro- bustness and broad applicability of our proposed method, we applied it to the DeepSeek 7B architecture. The results, pre- sented in Table 1, demonstrate that our method consistently enhances model performance across different pretraining stages and model architectures. This highlights the univer- sal generalizability of our approach beyond the Baichuan2 family, establishing its effectiveness as a model- optimization strategy.
CMMLU
56.78/56.78
56. (6.62/66.72 72
56.97/56.91
MMLU
54.82/54.54
16/16/54,54
54.56/55.29
GSM8K
21.92/23.65
22.44/24.34
24.32/24.40
A(J)
2.01
2.38
0.87
models ranging from 70M to 6.9B parameters. We assessed
performance on PIQA, WinoGrande, SciQ, and ARC-Easy datasets, with detailed PIQA results presented in Table 7.
Key Findings. Across all parameter sizes, our method consistently outperforms baseline merging approaches. For the Pythia 70M model, our method achieves PIQA score of 60.18 %, surpassing the best-performing baseline (Reg- Mean) by 1.22%. Similar improvements are observed across larger models:
4.2. Generalization to Unseen Domains
To rigorously assess the generalization capabilities of merged checkpoints to unseen domains, we evaluated the out-of-domain (OOD) performance of various merging methods. Although merging weights were determined using the C-Eval dataset (in-domain, IND), we tested the merged models on three diverse OOD datasets: CMMLU, MMLU, and GSMSK.
(a) Pythia 410M: Our method achieves 68.85% %, outper- forming the best baseline by 0.53 %.
(b) Pythia 1.4B: A score of 71.84% is achieved, which is 0.80% higher than RegMean.
Table 2 summarizes the results, revealing two critical in- sights:
(c) Pythia 2.8B: Our method attains 75.71% %, an improve- ment of 0.74% over the best baseline.
(a) Cross-Lingual Generalization:
Despite determining merging weights using a Chinese dataset (C-Eval), the merged checkpoints consistently perform well En- glish datasets such as MMLU and GSM8K. This indi- cates that merging pretraining checkpoints in parameter space preserves the models' ability to generalize across languages effectively.
(d) Pythia 6.9B: A consistent improvement is observed with 76.56 %2 surpassing the best baseline by 0.12 %.
These results demonstrate that our merging method main- tains effectiveness across diverse model scales, ensuring robust performance improvements regardless of the under- lying LLM architecture size.
(b) Superior Stability and Cross-Domain Performance: Our proposed method outperforms Greedy Soup and Fisher Weighted Averaging by exhibiting the small- est average absolute difference (A) between IND and OOD performance. Specifically, our method achieves A of 0.87, compared to 2.01 and 2.38 for Greedy Soup and Fisher Weighted Averaging, respectively. This demonstrates that our Bayesian optimization approach identifies merging weights that are more optimal across different domains.
4.4. Efficiency in Determining Optimal Merging Weights
To evaluate the efficiency of our proposed method in iden- tifying optimal merging weights, we compared it against various search strategies, including Random Search (Zabin- sky et al., 2009), Greedy Search, and Grid Search. Table 10 presents the results of merging Baichuan2- --ahhunnnn---BBBB-- with Baichuan2-2200B across multiple datasets using different search methods.
4.3. Performance Across Different Model Sizes
To evaluate whether our proposed merging method main- tains effecttivneess across LLMs with varying parameter sizes, we conducted comprehensive experiments on Pythia
Performance Analysis. Our Bayesian Optimization (BaycsOpt) based method consistently achieves the highest scores across all evaluated datasets:
7

---

Maximizing Checkpoint Value with Bayesian Optimization
Table 3: Scalability analysis of checkpoint merging methods across Pythia models varying parameter sizes (70M to 6.9B) evaluated on the PIQA dataset. The table demonstrates the consistent effectiveness of our Bayesian optimization approach across different model scales, with improvements over the best baseline methods. The results validate that our merging strategy maintains its effectiveness independent of model size, ensuring robust performance improvements across the entire spectrum of modern LIM architectures.
PIQA (5-shot)
Pythia 70M
Pythia 410M
Pythia 1.4B
Pythia 2.8B
Pythia 6.9B
Training step-142000
58.00
68.06
70.95
74.81
75.30
Training step-143000
58.54
68.06
70.89
74.31
76.44
Unitrom Soup
58.71
68.06
70.78
74.97
75.68
Greedy Soup
58.71
68,06
70.57
74.76
76.01
Fisher
58.69
68.14
70.87
74.65
75,94
RegMean
58.96
68.32
71.04
74.72
75.87
Ours
60.18 (+1.22)
68.85
(40.53)
71.84 (+0.80)
75.71 (+0.74)
76.56 (40.12)
Table 4: Efficiency comparison of different search strategies for determining optimal merging weights when combining Baiehuan2- 1980B with Baichuan2-22008 checkpoints. The evaluation compares Random Scarch, Greedy Search, Grid Scarch, and our Bayesian Optimization approach across four benchmark datasets. Our method consistently outperforms all baseline search strategies, achieving improvements over the best alternative methods, while requiring significantly fewer function evaluations due to the principled exploration- exploitation balance of Gaussian Process-based optimization.
Baichuan2-119088&22200 Random Search
Greedy Search
Grid Search
Ours
C-Fvvl((-shht)
$5.64
$5.49
55.51
56.73(+ L.09)
CMMLU((-shot)
55.67
56.74
55.51
57.05(+0.31)
MMLU(S shot)
54.16
54.35
54.77(+0.32)
GSMSK(4-shot)
20.56
21.83
20.85
22.17(+0.34)
Average
46.51
47.13
46.56
47.68(+0.55)
(a) C-Eval (5-shot): 56.73% (+1.09%) compared to Ran- dom Search (55.64%).
validation data. Performance remains stable across dataset fractions (56.61 % to 56.08%) indicating that our Bayesian optimization approach can effectively determine optimal merging weights with modest validation requirements, en- hancing practical applicability.
(b) CMMLU (5-shot): 57.05% Greedy Search (56.74%)
()) outperforming
(c) MMLU (5-shot): 54.77% % (+0.32%) surpassing
5.2. Impact of Merging Weight Search Space Size
Greedy Search (54.45%).
Our merging strategy involves hyperparameter a controlling the merging weight search space size, defined in Eq. (2). We assess a's impact by setting values 0.5, 0.7, and 0.9.
(d) GSM8K (4-shot): 22.17%
(+0.34%) exceeding
Greedy Search (21.83%).
Figure 4 illustrates performance as function of search space size. Key observations include:
Overall, our method achieves an average score of 47.68 %, outperforming Random Search (46.51 %), Greedy Search %.). ), and Grid Search (46.56% %) This demonstrates that our Bayesian optimization approach effectively explores the search space and iteratively refines the search identify optimal merging weights.
(a) Optimal Values: Setting to 0.5 0.7 yields optimal performance, converging merging weights within (0.87, 0.89).
(b) Excessive Constraint Detriment: Seuting to 0.9 leads to noticeable accuracy decline, indicating overly restrictive search spaces dilute optimization effective- ness.
5. Ablation Study
5.1. Impact of Held-out Dataset Size on Checkpoint Merging
(c) Performance Gap Consideration: For significant per- formance gaps between checkpoints, narrower search spaces (lower a) prove beneficial. Conversely, for similar-performing checkpoints, broader search spaces allow greater flexibility in weight allocation.
Following established practices (Wortsman et al., 2022; Matena & Raffel, 2022), our proposed method requires validation set to determine optimal merging weights. We investigate the impact of held-out dataset size variations by extracting different fractions of the C-Eval validation data and testing merging performance for Baichuan2-22008 and Bai2huan2-2420B checkpoints.
5.3. Empirical Analysis of Multi- Calti-Checcpoint Merging
Building upon our pilot experiments, we investigated perfor- mance effects when combining varying numbers of check- points during pretraining. Using the C-Eval dataset, we employed the greedy soup strategy (Wortsman al., 2022)
Results presented in Table 5 demonstrate that held-out dataset size exerts minimal influence on our method's 's ef- ficacy, maintaining robust performance even with limited
8

---

Maximizing Checkpoint Value with Bayesian Optimization
Table Robustness analysis of our checkpoint merging method with respect to held-out validation dataset size. The experi- ment evaluates merging performance of Baichuan2- 22002 and Baichuan2-24200 checkpoints on the C-Fval dataset using differ- ent fractions (1/4, 1/2, 3/4, full) of the validation data for weight optimization. Results demonstrate minimal sensitivity to dataset size variations, indicating that our Bayesian optimization approach can effectively determine optimal merging weights even with lim- ited validation data, enhancing the practical applicability of our method.
$6.65
56,00
56,02
55
$4.00
55.44
$4.65
52,65
$3.00
11.09
52.73
$1.76
50.62
Accuracy
50.55
50
40.47
49.4
49.66
C-Eval
1/4
1/2
3/4
1
3/.40
Ours
56.61
56.08
55.80
56.23
25
checkpoints before Harging
23.85
Merging Checkpoints
22.09
Merging Three Checkpoints
56.0
55.87
55.90
22
Merging Four Checkpoints
55.63
1980B
2208
440B
6608
8808
1100B
1320B
1540B
17608
1980B
22008
2420B
55.5
55.30
Accuracy
55.0
Figure 5: Empirical comparison of merging strategies across different numbers of adjacent checkpoints using the Greedy Soup method the C-Eval dataset. The analysis compares performance when merging two, three, and four consecutive Baichuan2 checkpoints across various training stages (200B to 2640B tokens). Results demonstrate that pairwise merging consis- tently outperforms multi-checkpoint combinations, with diminish- ing returns checkpoints are included.
54.5
54.40
1760B
54.0
53.5
0.5-1
0.7-1 Hyperparameter Range
0.9-1
various experiments, we find that: our proposed approach has the potential enhance pretraining, offering nearly a free lunch Besides superior performance, the merged result still exhibits strong generalization capability across vari- ous domains, which means our proposed method does not compromise the generalization of pretraining checkpoints.
Figure 4: Effect of merging weight search space boundaries the performance of merged Baichuan2- 1-60B and Baichuan2- 1980B models evaluated on the C-Eval dataset. The figure illustrates how varying the lower bound parameter a (set to 0.5, 0.7, and 0.9) influences accuracy outcomes in our Bayesian op- timization framework. Results show that moderate search space constraints 0.5 or 0.7) yield optimal performance, while overly restrictive bounds (a = 0.9) lead to performance degrada- tion.
A Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant No. 62306087 and 62472121), the Natural Science Foundation of Shandong Province (Grant No. ZR2023QF154), Special Funding Program of Shandong Taishan Scholars Project and and wang' Research Fund (Grant No. 20240204).
to merge adjacent three or four checkpoints across different pretraining stages.
Results presented in Figure 5 reveal that pairwise merging consistently outperforms multi-checkkoint combinations. For instance, merging Baichuan2-13222 with Baichuan2- 1540B achieves 53.06 (+2.14). while merging three checkpoints (Baichuan2-1 100B, Baichuan2-13208, and Baichuan2-1540B) yields 51.76% (+0.84), and four check- points result in further reduced performance of 51.01 % (+0.09). This validates our pairwise merging strategy's 's com- putational efficiency and performance optimality.
CCF--angfoo "Yuan-
6. Conclusion
In this paper, to alleviate the huge computational cost pre- training LLM, propose merging checkpoints in the pre- training trajectory. Specifically, we first conduct some pilot experiments to explore the characters of checkpoint merg- ing. Then, based on the findings in the pilot experiments, we propose a method rooted Bayesian optimization to find the optimal or near-optimal merging weight. Through
9

---

Maximizing Checkpoint Value with Bayesian Optimization
Impact Statement
Gi, J., Li, Y., Li, Y. K., Liang, W., Lin, E,, Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F,, Ma, S., Nie, X., Pei, T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang, P., Wang, S., Wang, Y., Wang. Y., Wu, T., Wu, Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang, D., You, Y.. Yu, S., Yu, X., Zhang, B., Zhang, H., Zhang, L.. Zhang, L.., Zhang, M., Zhang. M., Zhang, W., Zhang, Y., Zhao, C., Zhao. Y., Zhou, S., Zhou, S., Zhu, Q. and Zou, Y. Deepseek llm: Scaling open-source language models with longtermism, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
References
Akiba, T., Shing, M., Tang, Y., Sun, Q and Ha, D. Evo- lutionary optimization of model merging recipes, arXiv preprint arXiv:2403. 13187, 2024.
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H.,
O' O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Faiz, A., Kaneda, S., Wang, R., Osi, R., Sharma, P., Chen, F., and Jiang, L. Limcarbon: Modeling the end-to-end carbon footprint large language models, arXiv preprint arXiv:2309. /4393, 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Frazier, A tutorial on bayesian optimization, 2018.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, Measuring mas- sive multitask language understanding. arXiv preprint wrXi::2009.03300 2020.
Brochu, E., Cora, V. M., and De Freitas, N. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning, arXiv preprint arXiv: 1012.250,,, 2010.
Hochreiter, S. and Schmidhuber, J. Flat minima. Neural computation, 9(1): 1997.
Hou, L., Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., and Zhou, D. Token dropping for efficient BERT pretraining. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguisties (Volume I: Long Papers), pp. 3774-3784, Dublin, Ire- land, May 2022. Association for Computational Linguis- tics. doi: 10. 10.18663//1/222..ooooooogg / //aclanthologe wwtp//gg//22/.. acl -ong. 262.
Chen, H., Lundberg, S., and Lee, S.-I. Checkpoint ensem- bles: Ensemble methods from single training process. arXiv preprint arXiv: 1710.0023 2017.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023.
URL https:
Chowdhury, S. R. and Gopalan, A. On kernelized multi- armed bandits. In International Conference on Machine Learning, pp. 844-853 PMLR, 2017.
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., Fu, Y., Sun, M., and He, J. C-eval: A multi-level multi multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems, 2023.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: arXiv:/803..m 0803.05457, 2018
Jin, X., Ren, X., Preotiuc-Pietro, D.,, and Cheng, P. Data- less knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09884 2022.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems, 2021.
Jolicoeur-Martineauu A., Gervais, E., Fatras, K., Zhang, Y., and Lacoste-Julien, S. Population parameter averaging (papa). arXiv preprint arXiv:2304.03094 2023.
DeepSeck-AI, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z., He, Y., Hu, W., Huang, P., Li, E., Li,
Kim, D., Park, C., Kim, S., Lee, W., Song, W., Kim, Y., Kim, H., Kim, Y., Lee, H., Kim, J., et al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312. 12.121111110 15166, 2023.
10

---

Maximizing Checkpoint Value with Bayesian Optimization
Li, H., Zhang, Y., Koto, F., Yang, Y., Zhao, H., Gong, Y., Duan, N., and Baldwin," 'T. Cmmlu: Measuring massive multitask language understanding in chinese, 2023a.
Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishknn, P., Monaco, V., Morikawa, E., Mossing. D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L.., O' Keeff, C.,, Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, F.,, Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong. V., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C.. Real, F., Rimbach, K., Ross, C.,, Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schmurr, D., Schulman, J., Sel- sam, D., Sheppard, K., Sherbakov, T.. Shich, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl. I., Sokolowsky, B., Song. Y., Stau- dacher, N., Such, F. P.. Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M., Tillet, P., Tootoonchian, A., Tseng. E., Tuggle, P.. Turley, N., Tworek, J., Uribe, J. F.C. Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng. L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu. K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang. J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2023.
Li, P., Zhang, Z., Yadav, P., Sung, Y.-L., Cheng, Y., Bansal, M., and Chen, T. Merge, then compress: Demystify efficient smoe with hints from its routing policy. arXiv preprint arXiv:2310.013: 2023b.
Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., and Stoica, I. Tune: A research platform for dis- tributed model selection and training. arXiv preprint (rXiv:800.005111 2018.
Liu, Z., Cheng, S., Zhou, H., and You, Y. Hanayo: Har- nessing wave-like pipeline parallelism for enhanced large model training efficiency. In Proceedings of the Inter- national Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-13, 2023.
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. Advances in Neural Information Processing Systems, 35:1 35:17703-17716, 2022.
McAllester, D. A. Some pac-bayesian theorems. In Proceed- ings of the eleventh annual conference on Computational learning theory, pp. 230-234, 1998.
OpenAI, :, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. I.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji. S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum. J., Bello, I., Berdine, J., SomnaddettShhaiinee G., Berner, C., Bogdonoff, L.., Boiko, O., Boyd, M., Brakman, A.-L,, Brockman, G., Brooks, T., Brundage, M., Button, K. C.i T., Camphell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B. Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J.. Dai, Y., Decarcaux, C., Degry, T,, Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, Fulford, I., Gao, L.,, Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., GoneijoLopes, R., Gordon, J., Grafstein. M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J. Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu. K. Hu, S., Hu, X., Huizinga. J., Jain, S., Jain. S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, I.., Kim, J. W., Kim, C., Kim, Y., Kirchner, H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D.,
OpenAI, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2:13, 2023.
Qin, Y., Zhang, J., Lin, Y., Liu, Z., Li, P., Sun, M., and Zhou, J. Elle: Efficient lifelong pre-training for emerging data. arXiv preprint arXiv:2203.06311 2022.
Fulford,
Rajbhandari, S., Rasley, J., Ruwase, O., and He, Y. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Per- formance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Ramé, A., Vieillard, N., Hussenot, I.., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. Warm: On the benefits of weight averaged reward models, 2024.
Rasmussen, E. Gaussian processes in machine learn- ing. In Summer school on machine learning, pp. 63-71. Springer, 2003.

---

Maximizing Checkpoint Value with Bayesian Optimization
Ruder, S. and Plank, B. Learning select data for trans- fer learning with bayesian optimization. arXiv preprint 1iXiv::707.05246 2017.
Catt, E., Attaluri, N., Balaguer, J., Xiang, J., Wang, P., Ashwood, Z.., Briukhov, A., Webson, A., Ganapathy, S., Sanghavi, S., Kannan, A., Chang, M.-W., Stjemnrenn, A., Djolonga, J., Sun, Y., Bapna, A., Aitchison, M. Pejman, P., Michalewski, H., Yu, T. Wang, C., Love, J., Ahn, J., Bloxwich, D., Han, K., Humphreys, P., Sellam, T., Bradbury, J., Godbole, V., Samangooei, S., Damoc, B., Kaskasoli, A., Amold, S. M. R., Vasudevan, V., Agrawal, S., Riesa, J., Lepikhin, D., Tanbur,, R., Srinivasan, S., Lim, H., Hodkinson, S., Shyam, P., Ferret, J., Hand, S., Garg, A., Paine, T. I.., Li, J., Li, Y., Giang, M., Neitz, A., Abbas, Z., York, S., Reid, M., Cole, E., Chowdhery, A., Das, D.. Rogozinska, D., Nikolaev, V., Sprechmann, P., Nado, Z., Zilka, L.,, Prost, F., He, L.., Monteiro, M., Mishra, G., Welty, C., Newlan, J., Jia, D., Allamanis, M., Hu, C. H., de Liedekerke, R., Gilmer, J., Saroufim, C., Rijhwani, S., Hou, S., Shrivastava, D., Baddepudi, A., Goldin, A., Ozturel, A., Cassirer, A., Xu, Y., Sohn, D., Sachan, D., Amplayo, R. K., Swanson, C., Petrova, D., Narayan, S., Guez, A., Brahma, S., Landon, J., Patel, M., Zhao, R., Villela, K., Wang, L., Jia, W., Rahtz, M., Giménez, M., Yeung, L., Lin, H., Keeling, J., Georgiev, P., Mineu, D., Wu, B., Haykal, S., Saputro, R., Vodra- halli, K. Qin, J., Cankara, Z., Sharma, A., Fernando, N., Hawkins, W., Neyshabur, B., Kim, S., Hutter, A., Agrawal, P., Castro-Ros, A., van den Driessche, G., Wang, T., Yang. F., yiin Chang, S., Komarek, P., Mcllroy, R., Lučic, M., Zhang. G., Farhan, W., Sharman, M., Natsev, P., Michel, P., Cheng, Y., Bansal, Y., Qiao, S., Cao, K., Shakeri. S., Butterfield, C., Chung. J., Rubenstein, P. K., Agrawal, S., Mensch, A., Soparkar, K., Lene, K., Chung, T., Pope, A., Maggiore, L.,, Kay, J., Jhakra, P., Wang, S., Maynez, J., Phuong, M., Tobin, T., Tacchetti, A.. Trebacz, M., Robinson, K., Katariya, Y., Riedel, S., Bailey, P., Xiao, K., Ghelani, N., Aroyo, L.., Slone, A., Houlsby, N., Xiong, X., Yang, Z., Gribovskaya, E., Adler, J., Wirth, M., Lee, L.. Li, M.. Kagohara, T., Pavagadhi, J., Bridgers, S., Bortsova, A., Ghemawat, S., Ahmed, Z., Liu, T., Pow- ell, R., Bolina, V., linuma, M., Zablotskaaa,, P., Besley, J., Chung, D.-.., Dozat, T., Comanescu, R., Si, X., Greer, J., Su, G., Polacek, M., Kaufman, R. L., Tokumine, S., Hu, H., Buchatskaya, E., Miao, Y., Elhawaty, M., Sid- dhant, A., Tomasev, N., Xing, J., Greer, C., Miller, H., Ashraf, S., Roy, A., Zhang, Z., Ma, A., Filos, A., Besta, M., Blevins, R., Klimenko, T., Yeh, C.-K., Changpinyo, S., Mu, J., Chang, O., Pajarskas, M., Muir, C., Cohen, V., Lan, C. I.., Haridasan, K., Marathe, A., Hansen, S., Dou- glas, S., Samuel, R., Wang, M., Austin, S., Lan, C., Jiang, J., Chiu, J., Lorenzo, J. A., Sjösund, L. L.,, Cevey, S., Gle- icher, Z., Avrahami, T., Boral, A., Srinivasan, H., Sclo, V., May, R., Aisopos, K., Hussenot, L., Soares, L. B., Baumli, K., Chang, M. B., Recasens, A., Caine, B., Pritzel, A., Pavetic, F., Pardo, F., Gergely, A., Frye, J., Ramasesh, V. Horgan, D., Badola, K., Kassner, N., Roy, S., Dyer,
Sakaguchi, K., Bras, R. L.., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909.0805 08053,, 2019.
Simpson, E., Gao, Y., and Gurevych, I. Interactive text ranking with bayesian optimization: A case study community qa and summarization. Transactions of the Association for Computational Linguistics, 8:759-775, 2020.
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. Gaussian process optimization in the bandit set- ting: No regret and experimental design. arXiv preprint arXiv:0912.39995, 2009.
Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in nlp. arXiv preprint IIX::I996600224.4 2019.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.- Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Mil- lican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lill- icrap, T., Lazaridou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Hennigan, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Doherty, R., Collins, F.,, Meyer, C., Ruther- ford, E. Moreira, E., Ayoub, K., Goel, M. Tucker, G., Piqueras, E., Krikum, M., Barr, I., Savinov, N., Danihelka, I. Roelofs, B., White, A., Andreassen, A., von Glehn, T., Yagati, L., Kazemi, M., Gonzalez, L., Khalman, M., Sygnowski, J., Frechette, A., Smith, C., Culp, L., Proleev, L., Luan, Y., Chen, X., Lottes, J., Schucher, N., Lebron, F., Rrustemi, A., Clay, N., Crone, P., Kocisky, T., Zhao, J. Perz, B., Yu, D., Howard, H., Bloniarz, A., Rae, J. W., Lu, H., Sifre, L., Maggioni, M., Alcober, F., Garrette, D... Barnes, M.. Thakoor, S., Austin, J..
G., Wong, W., Joshi, R., Chaabouni, R., Fatiha, D., Ahuja, A., Liu, R., Li, Y., Cogan, S., Chen, J., Jia, C., Gu, C., Zhang, Q., Grimstad, J.. Hartman, A. J.. Chadwick, M., Tomar, G. S., Garcia, X., Senter, E., Taropa, E., Pillai, T. S., Devlin, J., Laskin, M., de Las Casas, D., Valter, D., Tao, C., Blanco, I.., Badia, A. P., Reitter, D., Chen, M., Brennan, J., Rivera, C., Brin, S., Iqbal, S., Surita,
G., Labanowski, J., Rao, A., Winkler, S., Parisotto, E., Gu, Y., Olszewska, K., Zhang, Y., Addanki, R., Miech,
A., Louis, A., Shafey, L.. E.,, Teplyashin, D., Brown, G.,
12

---

Maximizing Checkpoint Value with Bayesian Optimization
E., Campos, V., Tomala, A., Tang. Y., Badawy, D. E., White, E., Mustafa, B., Lang, O., Jindal, A., Vikram, S., Gong, Z., Caelles, S., Hemsley, R., Thornton, G., Feng, F., Stokowiec, W., Zheng, C., Thacker, P., Çağlar Unlü, Zhang. Z., Saleh, M., Svensson, J., Bileschi, M., Patil, P., Anand, A., Ring. R., Tsihlas, K., Vezer. A., Selvi, M., Shevlane, T., Rodriguez, M., Kwiatkowski, T., Daruki, S., Rong, K., Dafoe, A., FitzGerald, N., Gu-Lemberg, K., Khan, M., Hendricks, L. A., Pellat, M., Feinberg, V., Coron-Kerr, J., Sainath, T., Rauh, M., Hashemi, S. H., Ives, R., Hasson, Y., Li, Y., Noland, F., Cao, Y., Byrd, N., Hou, L., Wang, Q. Sottiaux, T., Paganini, M., Lespiau, J.-B., Moufarek, A., Hassan, S., Shivakumar, K., van Amersfoort, J., Mandhane, A., Joshi, P. Goyal, A., Tung, M., Brock, A., Sheahan, H., Misra, V., Li, C., Rakićević, N., Dehghani, M., Liu, F., Mittal, S., Oh, J., Noury, S., Sezener, E., Huot, F., Lamm, M., Cao, N. D., Chen, C., Elsayed, G., Chi, E., Mahdieh, M., Tenney, I., Hua, N., Petrychenko, I., Kane, P., Scandinaro, D., Jain, R., Uesato, J., Datta, R., Sadovsky, A., Bunyan, O., Rabiej. D., Wu, S., Zhang, J., Vasudevan, G., Leurent, E., Alnahlawi, M., Georgescu, I., Wei, N., Zheng, I., Chan, B., Rabinovitch, P. G., Stanczyk, P., Zhang. Y., Steiner, D., Naskar, S., Azzam, M., Johnson, M., Paszke, A., Chiu, C.-C., Elias, J. S., Mohiuddin, A., Muhammad, F., Miao, J., Lee, A., Vieillard, N., Potluri, S., Park, J., Davoodi, E., Zhang, J., Stanway, J., Garmon, D., Karmarkar, A., Dong, Z., Lee, J., Kumar, A., Zhou, L., Evens, J., Isaac, W., Chen, Z., Jia, J., Levskaya, A., Zhu, Z., Gorgolewski, C., Grabowski, P., Mao, Y. Magni, A., Yao, K., Snaider, J., Casagrande. N., Suganthan, P., Palmer, E.,, Irving. G., Loper, E., Faruqui, M., Arkatkar, I., Chen, N., Shafran, I., Fink, M., Castaño, A., Giannoumis, I., Kim, W., Rybiński, M.. Sreevatsa, A., Prendki, J., Soergel, D., Goedeckemeyer, A., Gierke, W., Jafari, M., Gaba, M., Wiesner, J., Wright, D. G., Wei, Y., Vashisht, H., Kulizhskaya, Y., Hoover, J., Le, M., Li, L.., Iwuanyanwu, C., Liu, L., Ramirez, K., Khorlin, A., Cui, A., LIN, T., Georgiev, M., Wu, M., Aguilar, R., Pallo, K., Chakladar, A., Repina, A., Wu, X., van der Weide, T., Ponnapalli, P., Kaplan, C., Simsa, J., L., S., Dousse, O., Yang, F., Piper. J., Ic, N., Lui, M., Pasumarthi, R., Lintz, N., Vijayakumar, A., Thiet, L. N., Andor, D., Valenzuela, P., Paduraru, C., Peng. D., Lee, K., Zhang, S., Greene, S., Nguyen, D. D., Kurylowiez. P., Velury, S., Krause, S., Hardin, C., Dixon, L., Janzer, L., Choo, K., Feng, Z., Zhang, B., Singhal, A., Latkar, T., Zhang. M., Le, Q., Abellan, F. A., Du, D., McKinnon, D., Antropova, N., Bolukbasi, T., Keller, O., Reid, D., Finchelstein, D., Raad, M. A., Crocker, R., Hawkins, P., Dadashi, R., Gaffney, C., Lall, S., Franko, K., Filonov, E., Bulanova, A., Leblond, R., Yaday, V., Chung, S., Askham, H., Cobo, L. C., Xu, K., Fischer, F., Xu, J., Sorokin, C., Alberti, C., Lin, C.-C., Evans, C., Zhou, H., Dimitriev, A., Forbes, H., Banarse, D., Tung, Z., Liu, J., Omernick, M., Bishop, C., Kumar,
C., Sterneck, R., Foley, R., Jain, R., Mishra, S., Xia, J., Bos, T., Cideron, G., Amid, E.,, Piccinno, E., Wang. X., Banzal, P., Gurita, P., Noga, H., Shah, P., Mankowitz, D. J., Polozov, A., Kushman, N.. Krakovna, V., Brown, S., Bateni, M., Duan, D., Firoiu, V., Thotakuri, M., Natan, T., Mohananey, A., Geist, M., Mudgal, S., Girgin, S., Li, H., Ye, J., Roval, O., Tojo, R., Kwong, M., Lee- Seee-Thor..11 J., Yew. C., Yuan, Q. Bagri, S., Simopalnikov, D., Ramos, S., Mellor, J., Sharma, A., Severyn, A., Lai, J., Wu, K., Cheng, H.-T., Miller, D., Sonnerat, N., Vnukov, D., Greig, R., Beattie, J., Caveness, E., Bai, I.., Fisenschlos, J., Ko- rehemniy, A., Tsai, T., Jasarevic, M., Kong, W., Dao, P.. Zheng, Z., Liu, F., Yang. F., Zhu, R., Geller, M., Teh, T. H., Sanmiya, J., Gladchenko, E., Trdin, N., Sozanschi, A., Toyama, D., Rosen, E., Tavakkol, S., Xue, L., Elkind, C., Wooddma, O., Carpenter, J., Papamakarios, G., Kemp, R., Kafle, S., Grumina, T., Sinha, R.,, Talbert, A., Goyal, A., Wu, D., Owusu-Afriyie, D., Du, C.,, Thornton, C., Pont-Tuset, J., Narayana, P., Li, J., Fatchi, S., Wieting, J., Ajmeri, O., Uria, B., Zhu, T., Ko, Y., Knight, L.,, Héliou, A., Niu, N., Gu, S., Pang, C., Tran, D., Li, Y., Levine, N., Stolovich, A., Kalb, N.,
R., Goenka, S., Yustalim, W., Strudel, R., Elqursh, A., Lak- shminarayanan, B., Deck, C., Upadhyay, S., Lee, H., Dusenberry, M., Li, Z., Wang. X., Levin, K., Hoffmann, R., Holtooool-Riiiie Malmi, E., Mirylenka, D., Tan, Q. Koh, C., Yeganeh, S. H., Pöder, S., Zheng. S., Pongetti, F., Tariq, M., Sun, Y., Ionita, L., Seyedhosseimi, M., Tafti, P., Kotikalapudi, R. Liu, Z., Gulati, A., Liu, J., Ye, X., Chrzaszez, B., Wang. L., Sethi, N., Li, T., Brown, B., Singh, S., Fan, W., Parisi, A., Stanton, J., Kuang, C., Koverkathu, V., Choquette- Choo, C. A., Li, Y., Lu, T., Ittycheriah, A., Shroff, P., Sun, P., Varadarajan, M., Bahargam, S., Willoughby, R., Gaddy, D., Dasgupta, I., Desjardins, G., Coreroo, M., Robenek, B., Mittal, B., Albrecht, B., Shenoy, A., Moi- seev, F., Jacobsson, H.. Ghaffarkhah, A., Rivière, M., Wal- ton, A., Crepy, C., Parrish, A., Liu, Y., Zhou, Z., Farabet, C., Radebaugg, C., Srinivasan, P., van der Salm, C., Fid- jeland, A., Scellato, S., Latorre-Chimoto, F.,, Klimezak- Plucińska, H., Bridson, D., de Cesare, D., Hudson, T., Mendolicchio, P., Walker, L., Morris, A., Penchev, I., Mauger, M., Guseynov, A., Reid, A., Odoom, S., Loher, L.. Cotruta, V., Yenugula, M., Grewe, D., Petrushkina, A., Duerig, T., Sanchez, A., Yadlowsky, S., Shen, A.. Globerson, A., Kurzrok, A., Webb, L., Dua, S., Li, D., Lahoti, P., Bhupatiraju, S., Hurt, D., Qureshi, H., Agar- wal, A., Shani, T., Eyal, M., Khare, A., Belle. S. R., Wang, L., Tekur, C., Kale, M. S., Wei, J., Sang, R., Saeta, B., Liechty, T., Sun, Y., Zhao, Y., Lee, S., Nayak, P., Fritz, D., Vuyyuru, M. R., Aslanides, J., Vyas, N., Wicke, M., Ma, X., Bilal, T., Eltyshev, E., Balle, D., Martin, N., Cate, H., Manyika, J., Amiri, K., Kim, Y., Xiong, X., Kang, K., Luisier, F., Tripuraneni, N., Madras, D., Guo, M., Waters,
D., Bachem, O., Yue, S., Arora, S.,
13

---

Maximizing Checkpoint Value with Bayesian Optimization
A., Wang, O., Ainslie, J., Baldridge, J., Zhang, H., Pruthi, G., Bauer, J. Yang, F., Mansour, R., Gelman, J., Xu, Y, Polovets, G., Liu, J., Cai, H., Chen, W., Sheng, X., Xue, E., Ozair, S., Yu, A., Angermueller, C., Li. X., Wang, W. Wiesinger, J., Koukoumidis, E., Tian, Y., Iyer, A., Gurumurthy, M., Goldenson, M., Shah, P., Blake, M., Yu, H. Urbanowiez, A., Palomaki, J., Fernando, C., Brooks, K., Durden, K., Mchta, H., Momchev, N., Rahimtoroghi, E., Georgaki, M., Raul, A., Ruder, S., Redshaw, M., Lee, J. Jalan, K., Li, D., Perng, G., Hechtman, B., Schuh, P., Nasr, M., Chen, M., Milan, K., Mikulik, V., Strohman, T., Franco, J., Green, T., Hassabis, D., Kavukcuoglu, K., Dean, J., and Vinyals, O. Gemini: A family of highly capable multimodal models, 2023.
Yogatama, D., Kong, L., and Smith, N. A. Bayesian optimization of text representations, In Mårquez, L., Callison-Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natu- ral Language Processing, pp- 2100-2105, Lisbon, Por- tugal, September 2015. Association for Computational Linguistics. doi: 10. 12..86////D1D151255111 URL https: / //aclanthology..... wtw.:/yy..ggg0110000000
Yu, L., Yu, B., Yu, H., Huang, F,, and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as free lunch, 2024.
Zabinsky, Z. B. et al. Random search algorithms. Depart- ment of Industrial and Systems Engineering, University of Washington, USA, 2009.
Touvron, H., Lavril, Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023.
Utans, J. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pp. 133-138 Citeseer, 1996.
Wan, F. Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models, 2024.
Welbl, J., Liu, N. F,, and Gardner, M. Crowdsoure- ing multiple choice science questions. arXiv preprint arXiv: MeXiv:/707.. 1a07,00000000 2017.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gones,o-opes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned mod- els improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965-23999 23998. PMLR, 2022.
Yang, A., Xiao, B., Wang, B., Zhang, B., Bian, C., Yin, C., Lv, C., Pan, D., Wang, D., Yan, D., Yang, F., Deng, F., Wang, F. Liu, F., Ai, G., Dong, G., Zhao, H., Xu, H., Sun, H., Zhang, H., Liu, H., Ji, J., Xie, J., Dai, J., Fang, K., Su, L., Song, L., Liu, L., Ru, L., Ma, L.. Wang. M., Liu, M., Lin, M., Nic. N., Guo, P., Sun, R.. Zhang, T.,
Li, T., Li, T., Cheng, W. Chen, W., Zeng, X., Wang, X., Chen, X., Men, X., Yu, X., Pan, X., Shen, Y.. Wang, Y., Li, Y.. Jiang, Y., Gao, Y., Zhang, Y., Zhou, Z.. and Wu, Z. Baichuan 2: Open large-scale language models, 2023a.
Yang. E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02555, 2023b.
14

---

Maximizing Checkpoint Value with Bayesian Optimization
A. Bounds for Linear Checkpoint Merging
In this section, we establish more realistic and insightful theoretical bounds on the performance merged model checkpoints, Considering consecutive checkpoints e 1 and O from the training of large language model (LLM), define the merged parameters θ as convex combination of these checkpoints:
b. x, + (1 1,)01-1, d e la,
(13)
where or E [0, 1) specifies the minimum weight assigned to θ
Let (()) denote the expected performance metric (e.g., accuracy) of the model with parameters θ evaluated on validation dataset D. Our goal is to derive theoretical bounds on f(O,) that align with realistic properties of neural network loss landscapes and provide deeper insights into the effects of linear checkpoint merging.
A.1. Assumptions
To enhance the theoretical framework, we adopt assumptions that reflect the practical characteristics of neural networks more closely.
Assumption 1 (Smoothness of the Performance Function). gradient V f(O) is Lipschitz continuous with constant Lg > 0:
The performance function f(O) is differentiable, and its
[V.)) (f(e))l & Lg - Obll, Aoa. Os.
(14)
Assumption 2 (Non-Convexity and Quadratic Approximation). While f(O) is generally non-convex, in the neighborhood of 0,-1 and Or it can be approximated by quadratic Specifically, for O O. + A, where |||| is small, we have
f(0) ~ V A H,
(15)
where H, the Hessian matrix al O
Assumption 3 (Bounded Hessian). The eigenvalues of the Hessian matrices at θt-1 and θ are bounded:
Amiul I Ht, Hi-1 Amaxl,
(16)
where Amin 2 and Amax are constants, and is the identity matrix.
A.2. Proof Performance Bounds
Proof. Under the assumptions, we derive tighter bounds on f(θ,). The derivation broken down into the following steps:
Step 1: Quadratic Approximation from θ,,
Expand f(x)) around θ, using the quadratic approximation (15) with A = =,, o, (1- A,):
f(θ,) = f(0,) + Vf().) A H,4 f(()) + (1) (0.-1
O:) + - (0) H, (, -O,)
(17)
Step 2: Quadratic Approximation from θ,-1-
Similarly, expand f(θt) around 0,-1 with A = 0, 0 A/(O) 0.-1):
f(,)) = f()+-))++++ V/( A A A H,-14
- (0, 0,-1) 1)+ + ( θ.-1) Hr,1 O.-1).
(18)
Step Combining the Approximations.
15

---

Maximizing Checkpoint Value with Bayesian Optimization
Take convex combination of (17) and (18), weighting each by and A²) respectively, obtain an averaged approximation:
ʃ(O,) = +1-
(111)) - v.((0--)] (9t-1-00 //4 [22(0) - 0,-1) (0, - (--1) + (1 (0) 1
-
(19)
Note that 0:-1 0 -0, Oi-1).
Step 4: Bounding the Gradient Difference.
Using the smoothness assumption (14), bound the gradient difference:
[V.) V - T. O--1|1-
(20)
Therefore, the term [V)())) ()((- (0t-1 - O,) can be bounded as:
(V.) v/(O)-1)] (0t-1 - 0.) - [V)(())) v |0,- - 0
[V)(())) v 10, 0,-1
Lg](O: 0
(21)
Step Bounding the Hessian Terms.
Using (16), bound the quadratic terms:
(O, 1 - O:-1) S Amax 0.-1
14-1
(0: -.-1) T Hi (0, O:-1) $ Amax | 1
(22)
Step Final Bound.
Substitute the bounds from (21) and (22) into (19) to obtain:
s(ō.) 2 (., + - ())((-))))))
- >x(1 - 1|1 - - [X2 + (1 - 2x) Amax/ O.-1
(23)
Similarly, using the upper bounds the performance function, we approximate:
f(ō.) S (O,) + (1 (-
+ A1111 1)) + 1 + (1 2.) Amax 0.-112
(24)
Step 7: Combining Lower and Upper Bounds. Combining (23) and (24), the performance of the merged checkpoint satisfies:
((θ.) (.)( + (1 >x) + (Ax(1 X.)Lo + 1 (1 Mmmx)) |0, 0.-111
(25)
B. Effect of Checkpoint Merging on Convergence
In this section, we analyze how linear checkpoint merging influences the convergence behavior gradient- based optimization algorithms. Our goal is to provide a theoretical framework that explains the potential benefits merging on convergence rates and the attainment of optimal performance.
16

---

Maximizing Checkpoint Value with Bayesian Optimization
B.1. Convergence in Standard Gradient-Band Based Training
In standard gradient-based training, model parameters are iteratively updated to minimize loss function L( over parameter space θ IR.. The updates are typically given by:
θ = -
(26)
where 7) > 0 is the learning rate, and VL(O;,1) is the gradient of the loss function evaluated at O,-1- The convergence behavior depends on factors such as the properties of the loss function (e.g., convexity, smoothness), the optimization algorithm, and hyperparameters like the learning rate.
= f(("). ( The goal of Let O" denote a local or global minimizer of L()), corresponding to the optimal performance f" = training is to find @* or parameters close to
B.2. Impact of Checkpoint Merging on Convergence
Checkpoint merging introduces mechanism whereby parameters from different iterations are linearly combined. This process can impact convergence in several ways. By interpolating between consecutive checkpoints, merging can effectively perform larger steps in parameter space, potentially accelerating convergence under certain conditions. Merging can also smooth out fluctuations in the optimization trajectory caused by stochastic gradients, leading to more stable convergence. Additionally, merging may help the optimization trajectory enter flatter regions of the loss landscape, which are associated with better generalization performance. Linear combinations of parameters can aid in escaping saddle points or shallow local minima by moving the parameters regions with lower loss.
B.3. Convergence Analysis
We analyze the impact of checkpoint merging on the convergence of the loss function and the performance metric f(O).
Consider the following setting: let { be the sequence parameters obtained from standard gradient updates. At each step t, we merge checkpoints to obtain θ via
6, = 0 + -1- O---11 e 1), a 10.
(27)
We define the performance improvement due to merging as
ff f(0,)
(28)
Our goal is to analyze f and understand how it influences convergence towards the optimal performance f"
B.3.1. ASSUMPTIONS
To facilitate the analysis, make the following assumptions:
Assumption 4 (Smoothness of the Loss Function). The loss function L( is twice continuously differentiable, and its
gradient VL(O) is Lipschitz continuous with constant L > 0:
[VL( -V())) < L|
Obll, v Os.
(29)
Assumption (Polyak-t.ojasiewice (PL) Condition). The loss function satisfies the PL condition, i.e., there exists a constant μ > 0 such that
[IVL 2 p (L(O) IT), AO.
(30)
where I, I[(*) is the minimal loss value.
Assumption 6 (Bounded Gradient Norms). The gradients have bounded norms:
|[V)( I Guax: Vt.
(31)
17

---

Maximizing Checkpoint Value with Bayesian Optimization
B.3.2. ANALYSIS OF A/t
We analyze the performance improvement f due merging. Since f(@) is related to L()) (c.g., higher performance corresponds to lower loss), we express f()) function of (()). For the purpose of analysis, assume that f())
decreases monotonically with L()) so that decrease in loss corresponds to an improvement in performance.
Proof. Step 1:
Taylor Expansion of the Loss Function.
Consider the Taylor expansion of L(O) at 6, for AO θ -,
L(O,) L(O.) + (L(O.) (Ô, 0t) + -.00.-0) O.) + R
(32)
where R represents the higher-order remainder terms.
Since θ is convex combination of O, and ④.-1, we have
= (1-
(33)
Substituting into the Taylor expansion, we get
= (L 01)
(1 (O)1 [(()))((0 -O,) R.
(34)
Step 2: Bounding the Remainder Term.
Under the assumption of Lipschitz continuity of the Hessian (i.e., the third-order derivatives are bounded), the remainder term can be bounded as:
M |0. -
|R
1,
(35)
where M is the Lipschitz constant for the Hessian. For sufficiently small ||0, - -0- the remainder term becomes negligible.
Step 3: Expected Improvement.
Taking expectations over the stochasticity in ①.-1 and O (due to stochastic gradients), and neglecting the remainder term, we have
[[[( ~ ( - (0
{1 A,) 3E (0,- (.).) O.)]
(36)
The term E (VL(O.)((O..) O can be related to the expected progress in gradient descent. Step 4: Relation to Gradient Descent Steps.
From the gradient update (26), we have
- -)))())
(37)
Assuming VL(O,-1) VL(O,) for small learning rates or smooth loss landscapes, we can write
θ -, ~
(38)
Substituting back into (36), we obtain
[[L(O:)) E[L(O.)] - (1
X.)3 (VL(O)) (V.()) [=L[O.)VL(..]]]
(39)
Step 5: Simplifying the Second-Order Term.
Assuming that v²I( is positive semi-definite (which holds for convex functions and in regions near local minima), we have
VL(),) (V)6)TTVV(()VVL(()) 2 Amin VL()))1
(40)
where Amin
0 is the smallest eigenvalue of (² Substituting back, get
((,( < [[L(O.)] - (.) + 1 (1 X1) Amax Alxx)
(41)
18

---

Maximizing Checkpoint Value with Bayesian Optimization
where max 2 0 the largest eigenvalue of (O,))))
Step 6: Net Expected Improvement.
Combining terms, the net expected reduction the loss due to merging is
[[ L( 2 7(1 Xx) (1 1 - Amxx B|| (()))
(42)
For sufficiently small learning rates and (1 - X values, the term in parentheses is positive, ensuring a net reduction the expected loss.
Step 7: Improvement in Performance.
Assuming that the performance metric f(@) improves as the loss decreases, we have
B[f(ū.)] 2 [[)():) + &c.
(43)
where & corresponds to the expected improvement in performance due the reduction in loss.
B.3.3. CONVERGENCE TOWARDS OPTIMAL PERFORMANCE
Under the PL condition (30), we can relate the squared gradient norm to the suboptimality loss:
|VL())]|| 2 2p (L(O,) L*).
(44)
Substituting into (42), we obtain
O(L(O.) L( 2mp(1 Xx) (1 (1- Ae) ([)(O,) -']..
(45)
This shows that the expected reduction in loss is proportional to the current suboptimality E [L(),) - L*), indicating that the merging process influences the convergence rate.
Step 8: Convergence Rate. Defining
Pt = 1==1-2appe 12pp11 -x) 21) (1-1001-0 5 dm)x)
(46)
have
ElL( - L- $ E[)( - --
(47)
For convergence, we require P: 1. Since n7, μ. Amax, and Ar) are positive, this condition can be satisfied with an appropriate choice of and \
Step 9: Accumulated Performance Improvement.
Over multiple merging steps, the accumulated suboptimality after T' steps is
E[ - - < II Pt (L(9)) - L*).
(48)
Assuming P: p (constant), obtain exponential convergence:
E[L(,r) - ((
(L(O)) L*).
(49)
Similarly, the performance metric converges towards the optimal performance f*:
E f f(ōr) S (f* f(Oo))
(50)
where 7 < depends on / and the relation between /(9) and (()).
19

---

Maximizing Checkpoint Value with Bayesian Optimization
C. Acceleration through Gaussian Process Optimization
In this section, we analyze how Gaussian Process (GP) optimization accelerates convergence to the optimal merging weight in checkpoint merging and potentially surpasses the reachability upper bound of standard training.
Gaussian Processes provide a flexible, non-parametric Bayesian approach to modeling unknown functions. In the context of checkpoint merging, model the performance function fAe) as GP:
f(x)) ~ GP (Ho(Ar), ((x,,)))
(51)
where (o(A,) is the mean function (often assumed to be zero), and (A,,, X₂) is the covariance (kernel) function encoding our assumptions about the smoothness and structure of [(Ae).
C.1. Efficient Convergence to the Optimal Merging Weight
Bayesian Optimization leverages Gaussian Processes to sequentially select ) values expected to improve the performance f(Ar), balancing exploration and exploitation. The selection is guided by an acquisition function a<Ac), such as Expected Improvement (EI), Upper Confidence Bound (UCB), or Probability of Improvement (PI). Under certain conditions, Bayesian Optimization with GPs can achieve convergence to the global optimum of ((x))-
By incorporating prior knowledge and uncertainty, Gaussian Processes enable Bayesian Optimization to identify promising regions of X with fewer evaluations compared to grid or random search. The cumulative regret Ry after 7 iterations, defined as the sum of the differences between the optimal performance fma and the performance at each iteration, is sublinear in T under mild assumptions. Specifically, for GP optimization with a bounded kernel function, we have
Ry O (vT(m++)) + 1)),
(52)
where "r is the maximum information gain from T observations (Srinivas et al., 2009; Chowdhury & Gopalan, 2017). The sublinear regret implies that the average regret RT /T decreases to zero as T - oc, ensuring convergence to the global optimum.
C.2. Breaking Through the Reachability Upper Bound
Standard training may converge to suboptimal performance due to factors like local minima or limited exploration of the parameter space. GP-based Bayesian Optimization checkpoint merging can potentially surpass this limit. Gaussian Processes capture the function's 's behavior, including any non-convexities or multimodalities, enabling the discovery of X values that yield better performance than those found via convex combination alone. Checkpoints θ,- and O may have learned complementary features; optimal X can exploit these synergies to improve generalization and performance beyond the convex combination's ' average. The acquisition function directs the search toward regions where the GP predicts higher potential gains, even if these regions are not suggested by local convexity assumptions.
C.3. Proof of Convergence
We formalize the convergence properties of GP-hased checkpoint merging.
Theorem (Convergence of GP-based Checkpoint Merging). Under the assumptions of Lipschitz continuity and local convexity of (Ai), and assuming bounded observation noise, the GP-based Bayesian optimization approach for checkpoint merging converges almost surely to a merging weight Xi that maximizes (Ot), such that
((6(T)) f(θ;) T
lim
f().)-
(53)
ArE(o,1]
Proof. Step 1: Establishing Continuity and Existence of the Optimum. Under the Lipschitz continuity assumption, f(xe) satisfies
( - J(X)) S LP>,
X/ VA X3 E [a, 1],
(54)
with Lipschitz constant L > 0. This ensures that fAx) is continuous over the compact interval ja, 1). By the Extreme Value
Theorem, there exists X; e [a, 1] such that
f(x)) = max f(xx).
(55)
A.Ela,1|
20

---

Maximizing Checkpoint Value with Bayesian Optimization
Step 2: Posterior Convergence of Gaussian Processes.
Bayesian Optimization with GP priors updates the posterior distribution of f(xx) after each observation. The GP posterior mean pr (A2) and variance of (Ae) at iteration T incorporate all previous evaluations. As T - oc, the GP posterior converges to the true function f()e) at the observed points. Given that the observation noise bounded or zero, uncertainty (posterior variance) at these points decreases zero.
Step 3: Acquisition Function and Selection of Next Point.
The acquisition function (r,( selects the next point X (T+1) by balancing exploration and exploitation:
X (T+1) arg max or((A); AeE(o,1]
(56)
where or (Ax) depends on (Ae) and or (Ae). For example, the Upper Confidence Bound (UCB) acquisition function is + UCB (Xx) = + Bror(A)),
0T
(57)
with 37 0 controlling the trade-off' between exploration and exploitation.
Step 4: Regret Analysis.
Under suitable choices of 3r, the cumulative regret Rr of the GP-UCB algorithm satisfies
T' Ry = Σ (1(A))
(v
(58)
where "r is the maximum information gain from 7 observations, typically logarithmic in T (Srinivas et al., 2009; Chowdhury & Gopalan, 2017). A sublinear cumulative regret implies that
lim Toc T
Rr = 0,
(59)
meaning the average regret per iteration decreases to zero, and the sequence (1, ( approaches X
Step 5: Almost Sure Convergence to the Optimal Weight.
By the convergence properties of GP-UCB algorithms, have
lim x(T)) X almost surely. T0000
(60)
Therefore, the performance of the merged model converges to
lim [(((1)) = f(x7), T-
(61)
which is the maximum achievable performance via merging in the interval [a, 1]. If ficcal is achievable within the merging interval- that if there exists X; such that f(})) fideal tten
lim (()(T) = fideeal). T
(62)
D. Impact on Generalization and Derivation of Generalization Bounds
In this section, we analyze how linear checkpoint merging affects the generalization performance of neural networks. We derive theoretical bounds that quantify this impact, leveraging the PAC-Bayesian framework. This analysis provides a deeper understanding of the benefits of checkpoint merging on generalization and offers thcoretical insights into its practical advantages.
21

---

Maximizing Checkpoint Value with Bayesian Optimization
D.I. Generalization Improvement through Flat Minima
Flat minima in the loss landscape are associated with better generalization performance because small perturbations in the model parameters lead to minimal changes in the loss function (Hochreiter & Schmidhuber, 1997). Checkpoint merging facilitates convergence to flatter regions by averaging parameters from different checkpoints, effectively smoothing the loss landscape and enhancing the model's robustness to unseen data.
By merging checkpoints -,- and o to obtain θ, we perform parameter averaging, which can be interpreted as moving towards flatter minima. This process reduces the model's sensitivity to parameter perturbations and contributes to improved generalization.
D.2. PAC-Bayesian Generalization Bound
We employ the PAC-Bayesian framework to derive generalization bound for the merged model +,, The PAC-Bayesian theorem provides probabilistic bounds on the generalization error of stochastic classifiers based on the Kullback-Leible (KL) divergence between posterior distribution Q over hypotheses (models) and prior distribution P.
D.2.1. DEFINITIONS
Let:
H. be the hypothesis space (set of all possible model parameters θ).
P be prior distribution over H.
Q be posterior distribution over H after observing the data.
e(O,
z) be the loss incurred by hypothesis θ on data point 2.
D be the data distribution.
S {= be the training consisting of i.i.d. samples from
(Q) Ee Ee-oEE-DI((()) 2)] be the expected loss (true risk) of Q.
Ls(Q) = 1e- Ei= ((O, z:) be the empirical loss (empirical risk) of Q.
D.2.2. PAC-BAYESIAN GENERALIZATION BOUND
Theorem (PAC-Bayesian Generalization Bound). For any prior distribution P over H. any 8 E (0, and any distribution D over the data, with probability at least 1 over the choice of the training set S, the following holds for all posterior distributions Q over H:
DKL (QI)P) In
2 (
L - (s(Q) +
(63)
2n
Proof. We provide step-by-step proof of the PAC-Bayesian generalization bound, following the methodology in McAllester (1998).
Step 1: McAllester's PAC-Bayesian Inequality
McAllester's PAC-Bayesian inequality states that for any E (0, 1), with probability at least - å over the choice of the training set S:
DkL(Q|P) + In
2y
Ee-q [LD(O)] s Ee-q [Cs(O)] +
(64)
2n
Donsker-Varadhan This inequality can be derived using concentration inequalities and the Donsker- Varadhan change of measure formula. Step 2: Deriving the Bound
Let us consider the moment-generating function of the empirical loss:
22

---

Maximizing Checkpoint Value with Bayesian Optimization
ES exp (s (LD(0) - Ls())))
exp
(65)
2n
for all I,, due to the boundedness of the loss function (assuming ((O,2) E [0, 1D. By integrating over Q and applying Fubini's theorem, we have:
82
ES Re-Q exp (8 (CD(O) - Ls( 2 exp
(66)
2n
Using Markov's inequality and applying a union bound over θ H, we can obtain a high-probability bound.
Step 3: Applying Lonsker-Varadhare Inequality
We utilize the Donsker- Donsker-Varadhan change of measure inequality:
Berolo(O)) in (ex=p(exp(o(e1)) + DKL(Q) P),
(67)
for any measurable function d
Let d(0)= (Cr(e) Ls(O)), then:
Ee-o (s (LD(0) - Cs(O))] In (Eg-r lexp (s (Co())) + DKI(Q P).
(68)
Since the expected value over P is bounded by exp 2m we have:
DKL(Q1 P) 2
Ea-q (Cv(O) -s(e))
(69)
Step 4: Optimizing over s
To obtain the tightest bound, we optimize the right-hand side with respect to s. Setting the derivative with respect to
3² DKL(Q
(70)
Os
8
Solving for 8, we find:
8" 2n (Q
(71)
Substituting back, obtain:
Re- (ED(O) Ls(O)) S DKL(Q [|
2n
8*
DkL + DKL (Q P)
(2nDKL(Q11 P)
2Dxx
(72)
Step Conversion to High-Probability Statement
Using Markov's ' inequality. we can convert the expectation into a high-probability statement. Specifically, for any E (0, 1),
with probability at least 8 over the choice of S:
23

---

Maximizing Checkpoint Value with Bayesian Optimization
2Dki + l (+)
Be- [CD(O)] Ee-q [Ls(()) +
(73)
2n
Simplifying, obtain:
DEL DKL P) + In (
Ee~ [CD(O)] s Ee-o [[s(
(74)
2n
Step 6: Finalizing the Bound
Adjusting for the discretization of the parameter space and covering numbers, as in McAllester (1998), we account for the additional term In n2) Thus, with probability at least 5, we have:
DKL (Q P) In 2//
+
(75)
2n
D.3. PAC-Bayesian to Checkpoint Merging
We apply the PAC-Bayesian bound to the merged model θ, obtained via checkpoint merging.
D.3.1. SETTING THE PRIOR AND POSTERIOR
We define the prior distribution P and posterior distribution Q as:
P P= P=N
(76)
where N(u, Σ) denotes a Gaussian distribution with mean j and covariance matrix E, and / is the identity matrix. Here, oZp and of are variance parameters controlling the spread of the distributions.
D.3.2. COMPUTING THE KI DIVERGENCE
Since P and Q are Gaussian distributions with covariance matrices If.. and I respectively, the KL divergence between Q and is:
(r(op2001 + (0,- 17) θ.) - - + del(
det(o
(K
no Q
||0 0,-1 1
In
(77)
0p
Assuming that of og 2² the divergence simplifies to:
202 |8 0.-1
(78)
24

---

Maximizing Checkpoint Value with Bayesian Optimization
D.3.3. EXPRESSING θ
Since θ is convex combination of + and O
6 - + (1-1111) --)
(79)
we have:
θ - 1 = X.(O, - O 0r-1), 0--1 Al110, -
(80)
Substituting back into the KL divergence:
X
(RI(Q P) 2x
| 01-1
(81)
D.3.4. EFFECT OF CHECKPOINT MERGING ON THE KL DIVERGENCE
Compared to directly using O, without merging, the divergence would be:
1 2n 10 0.-1112
(Ri(Qe, P)
(82)
Since d < 1, it follows that:
DKL(Q DKL P) < DKL(Qe, P).
(83)
Thus, checkpoint merging reduces the KI. divergence between the posterior and the prior, leading to a tighter generalization bound.
D.4. Combining Performance and Generalization
We aim to connect the empirical performance and the generalization bound to obtain an overall bound on the expected performance of the merged model on unseen data.
1. RELATION BETWEEN PERFORMANCE AND LOss
Assuming that higher performance corresponds lower loss (e.g., accuracy is inversely related error rate), we can define the performance metric f(O) as:
f()) = fmax L(
(84)
where fiax the maximum attainable performance, and I.(9) the generalization loss.
D.4.2. EXPECTED PERFORMANCE ON TEST DATA
Let:
frest (Q) Seenolfesste = fmax Lo(Q),
(85)
Jrain (Q) Ee-o [ Irain (9) Smax Ls(Q).
(86)
25

---

Maximizing Checkpoint Value with Bayesian Optimization
D.4.3. DERIVING THE COMBINED BOUND
From the PAC-Bayesian bound (63), we have:
5 +
(87)
where:
DKL || P) + in 22
(88)
2n
Substituting into (85):
fiest (Q)
faax (Q(
2 fmax - (Cs(Q) +
(Fmax Ls(Q)) -
Suain -
(89)
Thus, have:
DEL [)) + In (2) 22/
(Qs) 2 Sfrain((n
(90)
2n
E. Related Work
Model Merging in LLM: Model merging focuses on the unification of several models into one coherent entity, aiming to harness the collective strengths and mitigate the individual weaknesses of each model BalicceeurMaarinead et al., 2023),
and has recently emerged as significant trend in the research Large Language Models. detail, (Wortsman et al., 2022) proposes model soup to improve accuracy without increasing inference time by averaging weights multiple fine-tuned models. (Jin et al., 2022; Yu et al., 2024; Wan al., 2024) investigate the problem of merging individual LM fine-tuned on different datasets to obtain a single model that performs well both across all dataset domains obtain capabilities. (Ramé et al., 2024) proposes using model merging to obtain reliable and robust reward model in RLHF However, we find that conducting model merging during pretraining receives little attention.
Bayesian Optimization NLP: Bayesian Optimization (BayesOpt) can efficiently optimize objective functions that take long time evaluate and are widely applied in NLP. In detail, (Yogatama et al., 2015) leverage BayesOpt for Text Representations. (Ruder & Plank, 2017) learn data selection measures using BayesOpt in transfer learning for sentiment analysis and parsing. (Simpson et al., 2020) proposes using BayesOpt for community QA and summarization. demonstrating its superiority in tasks requiring nuanced feedback interpretation. Besides, (Brochu et al., 2010; Liaw et al., 2018) find that Gaussian process preference learning enables rapid, efficient inference, making it suitable for interactive applications requiring quick user feedback processing. this paper, unlike previous work, we use BayesOpt to obtain the merging weight for checkpoint merging in I.LM pretraining.
F. Performance on DeepSeek
In our initial experiments, we delve into the possibility of boosting performance through strategic checkpoint merging within the Baichuan pre-trained model, focusing on the checkpoint pairs 1540B and 1760B, along with 2200B and 2420B, in the advanced stages pretraining. This exploration
26

---

Maximizing Checkpoint Value with Bayesian Optimization
Dataset
Chekpoint-220E
Checkpoint
Uniform Soup
Greedy Soup
Fisher
RegMean
C-Eval
23.89
34.12
24.10
34.12
26.68
26.37
CMMLU
25.54
37.11
25.78
37.11
27.46
25.23
MMLU
23.85
33.29
33.29
23.15
23.33
GSM8K
6.82
9.10
8.04
9.10
8.16
5.46
Table 6: The results of merging baichuan2-220B with baichuan2-440B across various benchmark datasets.
WinoGrand (5-shot)
Pythia 70M
Pythia 410M
Pythia 1.4B
Pythia 2.8B
Pythia 6.9B
Training step-14
53.51
$7.77
63.85
Training step-143000
51.07
53.51
$7.38
60.93
63.61
Unifrom Soup
51.07
53.83
$6.99
61.09
63.61
Greedy Soup
$1.07
53.83
$7.06
60.85
63.77
Fisher
52.08
53.88
57.96
60.57
63.84
RegMean
51.97
53.76
58.03
60.89
63.55
Ours
52.35
(+0.10)
53.96 (+0.08)
58.72 (40.69)
62.04 (40.95)
64.46 (+0.61)
Table 7: The results of merging various parameter sizes of Pythia models using different merging methods on the WinoGrande datasets.
aims to uncover patterns of improvement that could applied across different models. However, our analysis has not extended to other models like DeepSeek. To bridge this gap, undertake detailed examination of DeepSeek by evaluating the merging of checkpoints 1800B and 2000B, assessing 100 merging weight points distributed evenly within the [0. 1] interval. Our objective is to ascertain if the trend of performance enhancement through merging observed Baichuan is also evident in other pre-trained models, specifically through the lens of weight combination efficacy.
0.66
Acsurscy (%)
0.400
44.36
2000B
The pivotal question guiding research is: Do similar patterns of performance improvement emerge in other pre-trained models, such DeepSeek, when applying strategic checkpoint merging, as observed in the Baichuan model?
0,45
43.05
1800B
06
02
B4
0.6
0.8
1.0
Merging Weight
Our research confirms this hypothesis, revealing that: Specifically, impressive 70% of the tested merging weight combinations for DeepSeek's checkpoints 1800B and 2000B result in performance enhancements.
Figure Validation checkpoint merging effec- tiveness on DeepSeek models: merging DeepSeek- 1800B and DeepSeck-2000B checkpoints with uni- formly sampled weights from [0,1] the C-Eval dataset. The performance curve demonstrates that of the tested merging weight combinations result in performance enhancements beyond the superior base model (DeepSeck-2000B).
This finding indicates that the strategy of merging checkpoints enhance performance is not unique the Baichuan model but is also applicable other pre-trained models like DeepSeek. The consistency of this pattern across different models highlights the potential of checkpoint merging as universally effective method for optimizing pre-trained model performance during their later training phases.
G. All the results of baichuan2-220B with baichuan2-4409 across various benchmark datasets.
As outlined in pilot experiments, performing weight merging checkpoints of LLMs during the early stages of pre-training leads to a degradation in performance. In the Table 6, we present our experimental results weight merging between baichuan2-2200 and baichuan2-440B models. We use Uniform Soup, Greedy Soup, Fisher, and RegMean to conduct merging and measure the post merging performance. Without exception, none of the outcomes show performance surpassing that of the superior base model mmichuan2-440B) This evidence suggests that conducting checkpoint merging in the early pre-training phase not a viable strategy.
27

---

Maximizing Checkpoint Value with Bayesian Optimization
SciQ(5-shot)
Pythia70M
Pythia410M
Pythial.4B
Pythia2.8B
Pythia6.9B
Training step- step-142000
142000
56.30
88.70
92.50
94.10
94.60
Training step-143000
58.20
89.40
92.30
93.60
94.90
Unifrom Soup
57.40
89.10
92.10
94.60
94.60
Greedy Soup
57.40
89.10
92.50
94.10
95,00
Fisher
58.10
88.70
92.90
93.90
94.50
RegMean
57.40
88.90
92.70
94.00
94.60
58.30 (+0.10)
89.20 (-0.20)
93.10 (+0.20)
94.80 (+0.70)
95.30 (+0.30)
Table 8: The results of merging various parameter sizes of Pythia models using different merging methods on the SciQ datasets.
ARC-Fasy(5-shot)
Pythia70M
Pythia410M
Pythial.4B
Pythia2.8B
Pythia6.9B
Training slep-142000
35.86
54.97
62.88
66.79
69.74
Training step- 143004
36.57
54.76
63.01
67.00
69.87
Unifrom Soup
36.70
54.67
63.05
67.00
70.12
Greedy Soup
36.70
54.67
62.96
67.17
69.95
Fisher
36.27
55.10
63.04
66.87
69.89
RegMean
36.14
55.21
62.98
67.01
70.22
Ours
36.58 (-0.12)
55.77 (40.56)
63.13 (40.08)
68.05 (+0.88)
70.82 (+0.60)
Table 9: The results of merging various parameter sizes of Pythia models using different merging methods on the ARC-Easy datasets.
Bacchuan2-1900-88222
EI
MPI
LCB
GP-hedge
C-Eval(S-shot)
$6.69
56.53
56. 14
56.73(+00.04)
CMMLU(5-shot)
56.88
56.89
56.50
57.05(++0116)
MMLU(5-shot)
54.60
54.54
54.62
54.77(+0.15)
GISMKK(4-shai)
22.17(+0.10)
Average
47.56
47.49
47.26
47.68(+0.12)
Table 10: The results of the ablation study for different acquisition strategies when merging Baichuan2-1980B with Baichuan2-2200B.
H. The table of different model size on Varying datasets
Acquisition Functions
We conduct ablation study different acquisition functions and find that using GP-hedge yields better results compared to individual acquisition functions. For instance, in the C-Eval (5-shot) task, GP-hedge achieves the highest performance with score of 56.73%, which is an improvement of 0.04% over the best individual acquisition function, EI, which scores 56.6% 69%... Similarly, in the CMMLU (5-shot) task, GP-hedge outperforms the others with a score of 57.05%, marking an improvement of 0.16% over the best individual acquisition function, MPI, which scores 56.89%. For the MMLU (5-shot) task, Ge-hedge again shows superior performance with score of 54.77%, surpassing the best individual function, LCB, by 0.15%. Lastly, in the GSMSK (4-shot) task, GP-hedge scores 22.17%, improvement of 0. over EI, which scores 22.07%. On average, GP-hedge shows significant improvement with an average score of 47. 47.68% which is 0.12% higher than the best individual acquisition function, EI, which averages 47.56%.
J. Pilot Experiments performance on CMMLU Dataset
In this section, we describe the performance merging checkpoints on the CMMLU dataset.
In accordance with the research question outlined in pilot experiments, we extended our analysis to the CMMLU dataset to corroborate the findings from the C-Eval dataset. Utilizing the same methodology, we evaluated all possible pairwise merging, totaling 55 combinations (Ci,). Employing the greedy soup strategy outlined by (Wortsman al., 2022), checkpoints sequentially incorporated into the soup if they demonstrate improvement in accuracy the development data.
The analysis yields observations that are consistent with those obtained from the C-Eval dataset, reinforcing the generality of our findings. Specifically, we observe that:
28

---

Maximizing Checkpoint Value with Bayesian Optimization
(1) Adjacent checkpoint merging yields superior performance: Echo- ing the C-Eval dataset's findings, merging two checkpoints from consecutive training phases in the CMMLU dataset generally led to performance enhancements over individual checkpoints. Notably, merging Baichuan2-19800 with Baichuan2-22008 achieved an ac-
2230 4000 6028 8008 11208 13208 13400 31008 L3906 2000 23009
curacy of 56.57% on the CMMLU dataset, surpassing the 56.29% accuracy Baich0 when assessed independently. This only highlights the effectiveness adjacent checkpoint merging but also indicates significant improvement over the final checkpoint, Baichuan-2422 (with an accuracy of 36.78%), by elevating the test
60 00.33
13.00)
12.94
13.00
$3.55
52
$2.89
22.66
32-40
30-21
23.33
012.01
0.57
0.000
(3.8)
14.20
531
16355
accuracy by 0.10%.
+8.90
07.4
(2) Merging distant checkpoints leads to performance deterioration: In line with the C-Eval dataset observations, merging significantly dis- parate checkpoints, such as Baichuan2-2201 with Faichuan2-22000 resulted in a notable performance decline, with accuracy dropping to 25.26% the CMMLU dataset. This outcome closely mirrors the performance of the lesser-trained checkpoint, Baichuan2-220DB which has an accuracy of 25. 25.54%, underscoring the negative impact of merging widely separated checkpoints.
122 (22-971333
29.08
26.873
23.40
8010277
25.28
m
20.88
23-14
26.11
(6,20 cot 20122 24-200 www.a///2..0000
6508
Figure 7: Performance comparison of pairwise checkpoint merging using the Greedy Soup method on the CMMLU benchmark across all possible com- binations Baichuan2 checkpoints. This heatmap visualization corroborates findings from the C-Eval dataset, showing that adjacent checkpoint merging (diagonal region) consistently yields performance im- provements, while distant checkpoint combinations re- sult in substantial performance degradation.
The CMMLU dataset findings reinforce the C-Eval dataset's s results, highlighting the critical importance of strategic checkpoint merging for enhanced model performance across diverse datasets.
29


Reviews:
[{"review_id": "DnwZSk1Zb8", "reviewer": "Reviewer_egL2", "summary": "The paper demonstrates that averaging adjacent checkpoints leads to better downstream performance compared to using individual checkpoints. To determine the optimal weighting, the paper proposes using Gaussian process-based Bayesian optimization.\nThe proposed approach outperforms existing merging strategies on downstream tasks, including commonsense reasoning tasks and mathematical exam questions.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1: Clarification Regarding Figure 3**\n> \n\n**Our Response:**\n\nWe appreciate this observation. **Figure 3** is not a result of the Bayesian Optimization procedure. Instead, it provides an **empirical exploration** of how the merged model’s performance changes when we *uniformly sample* $\\lambda$ in $[0, 1]$. In other words, it is a “landscape” study (a brute-force sweep of $\\lambda$ ) to:\n\n1. **Observe Variability for Large Gaps:** When checkpoints differ substantially in performance, $\\lambda \\approx 1$ typically yields higher performance, because weighting the stronger checkpoint is beneficial.\n2. **Observe Smooth Behavior for Similar Checkpoints:** When checkpoints have closer performance levels, a broad range of  $\\lambda\\in[0,1]$ can give improvements, reflecting that interpolating between two similarly strong models often yields robust results.\n\nBecause **Figure 3** revealed that performance often worsens if one heavily weights a weaker checkpoint, our practical merging approach **restricts** $\\lambda\\in[\\alpha, 1]$ with $\\alpha>0.5$. Empirically, this domain ensures the more recent (and presumably stronger) checkpoint $\\Theta_t$  maintains a dominant contribution. Hence, **Figure 3** is purely illustrative—an exploratory “map” of $\\lambda$-versus-accuracy—while the actual search for $\\lambda$ is done by Bayesian Optimization on the narrower interval $[\\alpha,1]$.\n\n> **Q2: Justification for Eq. 10.**\n> \n\n**Our Response:**\n\n**Eq. 10** appears in our **PAC-Bayesian generalization bound**, where we assume\n$P \\ = \\ \\mathcal{N}(\\Theta_{t-1}, \\sigma_P^2 I),\n\\quad\nQ \\ = \\ \\mathcal{N}(\\widetilde{\\Theta}t, \\sigma_Q^2 I),$\n*and derive $D_{\\mathrm{KL}}(Q \\  | | \\  P)$*. While real neural networks often exhibit correlated parameter distributions, we rely on:\n\n1. **Analytical Tractability:** A diagonal covariance structure (or isotropic $\\sigma^2I$) keeps the KL divergence in closed form:\n$D_{\\mathrm{KL}}(Q || P) \\ = \\ \\frac{\\|\\widetilde{\\Theta}t - \\Theta{t-1}\\|^2}{2\\sigma^2}.$\nThis simplification is standard in many PAC-Bayes treatments and Bayesian NN approximations (e.g., “mean-field” approximations).\n2. **Boundedness:** Even if true covariances are not diagonal, using a diagonal approximation typically *overestimates* correlations (or lumps them into the diagonal variance), making the bound somewhat looser but still valid as an upper bound.\n3. **Consistency with Empirical Practice:** Prior works (e.g., [1]) frequently adopt similar diagonal or isotropic assumptions for theoretical clarity and to yield tractable bounds.\n\n> **Q3: Part of Uncertainty Bounds in Table 1.**\n> \n\n**Our Response:**\n\nWe have now run **multiple-seed** experiments (e.g., 5 random seeds) for each method and checkpoint merge. We report both the **mean** accuracy and **95\\% confidence intervals (CIs)**. Below is an illustrative example for C-Eval (5-shot):\n\n| **Model / Method** | **Accuracy** | **95% CI** |\n| --- | --- | --- |\n| **Baichuan2-2200B** | 55.65 | [54.66, 55.98] |\n| **Baichuan2-2420B** | 54.90 | [54.32, 55.41] |\n| **Uniform Soup** | 55.47 | [54.89, 56.01] |\n| **Greedy Soup** | 55.58 | [54.94, 56.23] |\n| **Fisher Weighted Averaging** | 55.77 | [55.14, 56.39] |\n| **RegMean** | 53.81 | [53.08, 54.53] |\n| **Ours (BayesOpt)** | **56.20** | [55.61, 56.90] |\n\nThese intervals show that differences are both *statistically* and *practically* meaningful. We will include such CIs for all key results in the revised manuscript.\n\n> **Q4: Miscellaneous Comments and Reference Corrections.**\n> \n\n**Our Response:**\n\nThank you for pointing out these reference issues. We have carefully reviewed and will correct all citation and reference errors.\n\n> **Q5: Details on the Optimization Trajectories in Bayesian Optimization**\n> \n\n**Our Response:**\n\nWe have **added figures** for model-fit plot at anonymous link: https://anonymous.4open.science/r/checkpoint-47F1/bayse.png that illustrate the Bayesian Optimization process:\n\n- **Posterior Mean & Confidence Intervals:** After each iteration, we plot the Gaussian Process posterior (mean + confidence interval) over $\\lambda$, and show which $\\lambda$ was selected next.\n- **Number of Iterations:** Typically, we only need about **10–15** iterations for a 1D problem to converge near an optimal $\\lambda$.\n\nRestricting $\\lambda\\in[\\alpha,1]$ with $\\alpha>0.5$ is a pragmatic choice. *If* users want more flexibility (e.g., to incorporate knowledge from an older checkpoint more strongly), they can lower $\\alpha$. In practice, $\\alpha\\approx 0.5$ balanced well the merging of older vs. newer checkpoints, **based on pilot experiments** that showed merges with $\\lambda<0.5$ rarely helped.\n\n**Ref:**\n\n[1] Dziugaite G K, Roy D M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data[J]. arXiv preprint arXiv:1703.11008, 2017.", "claims_and_evidence": "The central claims of the paper are to a) provide \"substantial benefits at minimal cost\" and b) achieve robust generalization performance.\nThe results show that the proposed approach does improve upon the baseline and outperforms simply selecting a single checkpoint. However, I would argue that the observed gains are relatively modest, and the claim of \"substantial benefits\" might be somewhat overstated.", "methods_and_evaluation_criteria": "Looking at Figure 3, I am left wondering what exactly is being optimized. For the two blue checkpoints, the objective function appears to exhibit high observation noise. By visual inspection, it seems somewhat random how to set \\lambda, as choosing a value close to 1 or close to 0 appears equally likely to yield similar results.\nFor the red checkpoints, there is a noticeable trend toward higher \\lambda; however, these values are not a) included in the search space \\lambda \\in [\\alpha, 1] where \\alpha is set to > 0.5 in the experiments, and b) effectively mean that only one of the checkpoints is selected according to Equation 2.\n\nI would like to see a model-fit plot, similar to Figure 1 in Frazier et al., to demonstrate that Bayesian optimization can effectively model the observed data.\nAdditionally, given that this is merely a simple 1-D problem, I do not fully understand why the lower bound for \\alpha needs to be carefully selected rather than simply setting it to 0.\n\nA Tutorial on Bayesian Optimization\nPeter I. Frazier", "theoretical_claims": "I am unsure whether the assumptions in Equation 10 are justified, particularly regarding the diagonal covariance matrix.", "experimental_designs_or_analyses": "Overall, the experiments appear well-structured and thorough. However, the results in Table 1 would be more convincing if they included uncertainty bounds.", "relation_to_broader_scientific_literature": "The key contribution of the paper to the broader literature is: a) demonstrating that merging adjacent checkpoints yields better results than merging more distant checkpoints, and b) formulating model merging as an optimization problem that can be addressed using Bayesian optimization.", "essential_references_not_discussed": "As far as I can tell, the paper addresses all relevant literature.", "other_strengths_and_weaknesses": "Strengths:\n\n    The paper provides valuable and easily interpretable visualizations, such as Figure 2, which compares different checkpoints.", "other_comments_or_suggestions": "- Line 141: The reference appears to be incorrect—should this refer to Figure 3?\n- Line 340, right column: Incorrect reference—should be Table 4.\n- I'd suggest that the authors extend the results in Table 4 to also include plots that shows the optimization trajectories of Bayesian optimization compared to for example random search as it is common practice in the Bayesian optimization literature. This will also provide a better sense on the convergence speed of the proposed approach.", "questions_for_authors": "- What values of \\lambda does Bayesian optimization actually select?\n- How many iterations are performed with Bayesian optimization?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBMrZaJlx4", "reviewer": "Reviewer_Znoq", "summary": "**Overview**  \nThe paper introduces a novel checkpoint merging strategy aimed at enhancing the efficiency of large language model (LLM) pretraining. The central idea is to exploit intermediate checkpoints by forming linear combinations in the parameter space while optimizing the merging weight via Bayesian optimization. The paper’s key contributions are as follows:\n\n- **Algorithmic Contribution:**  \n  It presents an efficient pairwise checkpoint merging protocol (outlined in Algorithm 1) that effectively reduces a high-dimensional weight search problem to a manageable one-dimensional search. The approach utilizes Gaussian Process (GP) regression with acquisition functions such as EI, PI, and UCB along with a GP-hedge strategy.\n\n- **Theoretical Insights:**  \n  The work includes thorough theoretical analysis with convergence proofs, performance bounds using quadratic approximations of the loss landscape, and a derivation of PAC–Bayesian generalization bounds that explain how merging helps to reduce model variance.\n\n- **Empirical Findings:**  \n  Experiments spanning several models (e.g., Baichuan2, DeepSeek, and Pythia) across datasets like C-Eval, CMMLU, MMLU, and GSM8K demonstrate that merging adjacent checkpoints (but not those that are far apart) leads to performance improvements over baseline methods such as Uniform Soup, Greedy Soup, Fisher Weighted Averaging, and RegMean. Moreover, the proposed method shows enhanced both in-domain and out-of-domain generalization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:  Quadratic Approximation Validity (Equation (15)).**\n> \n\n**Our Response:**\n\n- Our derivation starts from the assumption that, for small perturbations, the performance function  $f(\\Theta)$  can be locally approximated by a quadratic form:\n$f(\\Theta) \\approx f(\\Theta_t) + \\nabla f(\\Theta_t)^\\top \\Delta + \\tfrac{1}{2} \\Delta^\\top H_t \\Delta,$\nwhere $\\Delta = \\widetilde{\\Theta}t - \\Theta_t$ *and $H_t$ is the Hessian of $f$ at $\\Theta_t$. This assumption leverages the smoothness $L_g$-Lipschitz continuity of  $\\nabla f$ and boundedness of the Hessian (i.e., $\\lambda_{\\min} I \\preceq H_t \\preceq \\lambda_{\\max} I$)*.\n- **Empirical Evidence:**\n    \n    Empirically, **Figures 1 and 3** in our paper illustrate smooth and nearly monotonic trends in performance with respect to the merging weight $\\lambda_t$. These trends indicate that, within a sufficiently small region between $\\Theta_{t-1}$ and $\\Theta_t$, the quadratic approximation is reasonable. In addition, our ablation studies show that—despite the nonconvexity of the loss landscape—the averaged performance follows the bounds predicted by our quadratic model (cf. **Eq. (17)–(25))**.\n    \n\n---\n\n> **Q2:**  Runtime Overhead of GP-Based Bayesian Optimization**.**\n> \n\n**Our Response:**\n\n- Our GP-based Bayesian optimization is designed to address an expensive, derivative-free objective. In our experiments for pairwise merging the search space is one-dimensional (i.e., $\\lambda_t \\in [\\alpha, 1]$). In a typical run we perform on the order of 10-15 evaluations. Empirical measurements show that such an optimization loop incurs an overhead of approximately 30–60 mins on **a** **single 4090 GPU**. In contrast, grid or random search methods typically require many more evaluations to reach comparable performance and may require **4–10**× more time per experiment.\n\n---\n\n> **Q3:  Relation to Gradient Smoothing and Sharpness-Aware Minimization.**\n> \n\n**Our Response:**\n\n- We acknowledge that recent studies on gradient smoothing and sharpness-aware minimization have focused on explicitly encouraging flat minima during training. Our approach to checkpoint merging can be viewed as complementary. By averaging weights from adjacent checkpoints, we effectively smooth over transient sharp local minima, thereby biasing the final model toward flatter regions of the loss landscape.\n- In our convergence and PAC–Bayesian analyses, smoother loss landscapes (i.e., flatter minima) correspond to tighter generalization bounds. Mathematically, if one denotes model sensitivity as $\\| \\nabla^2 L(\\Theta) \\|$, then averaging (i.e., merging) reduces the effective curvature:\n$\\widetilde{\\Theta}t = \\lambda_t \\Theta_t + (1-\\lambda_t)\\Theta{t-1} \\quad \\Rightarrow \\quad \\| \\widetilde{H}t \\| \\leq \\lambda_t \\| H_t \\| + (1-\\lambda_t)\\| H{t-1} \\|,$\nthereby promoting flatness. We will revise the manuscript to explicitly contrast our method with these related works and add references to the most recent literature in sharpness-aware minimization.\n\n---\n\n> **Q4:  Multi-Checkpoint Merging Versus Pairwise Merging.**\n> \n\n**Our Response:**\n\n- Our experiments (see **Figure 7**) reveal that while pairwise merging provides substantial improvements, extending the merging to three or four checkpoints often leads to performance dilution. We attribute this to the fact that the most recent checkpoints already capture the latest state of the model, whereas merging with older checkpoints may introduce bias or “stale” information.\n    \n    Furthermore, increasing the number of checkpoints increases the dimensionality of the search space exponentially—extending from a one-dimensional segment (two checkpoints) to a two-dimensional triangle (three checkpoints), and ultimately to a closed tetrahedron (four checkpoints)—which exacerbates the computational cost and complexity of the optimization process.\n    \n    We have conducted additional experiments employing Bayesian optimization for merging more than two adjacent checkpoints. The results are summarized in the table below:\n    \n    | Number of Checkpoints | C-Eval |\n    | --- | --- |\n    | 2 (Our original method) | 56.20 ± 0.52 |\n    | 3 | 56.27 ± 0.55($\\uparrow$ 0.07) |\n    | 4 | 55.34 ± 0.23($\\downarrow$ 0.86) |\n    \n    These results justify our decision to adopt pairwise merging as it achieves a favorable trade-off between performance gains and computational efficiency.\n    \n\n---\n\n> **Q5:  Generalization Across Different Architectures.**\n> \n\n**Our Response:**\n\n- Our empirical results indicate that the proposed merging method consistently improves performance across various architectures (i.e., Baichuan2, DeepSeek, and Pythia models). Although there are minor variations in the optimal merging weights—typically on the order of 0.1–0.7% accuracy—the overall trend of enhanced generalization and reduced variance is maintained.", "claims_and_evidence": "- **Main Claims:**  \n  - Merging intermediate checkpoints can notably boost LLM pretraining performance without adding much computational cost.\n  - Bayesian optimization is a practical and effective tool for determining near-optimal merging weights, outperforming conventional merging strategies.\n  - Models produced via this merging process exhibit superior generalization, as supported by PAC–Bayesian generalization bounds.\n\n- **Evidence Supporting the Claims:**  \n  - **Empirical Evidence:** Extensive tests—including pairwise and multi-checkpoint merging strategies—across various benchmarks back up these claims convincingly.\n  - **Theoretical Evidence:** The paper provides detailed proofs based on common assumptions (like Lipschitz continuity, bounded Hessian, and the Polyak–Łojasiewicz condition) that support the theoretical claims on convergence and generalization.\n  - **Comparative Analysis:** The systematic comparison against several existing merging techniques highlights consistent performance gains with the proposed approach.\n\n- **Question:**  \n  - **Location:** Section A, Equation (15) (Quadratic Approximation)  \n  - *Can the authors provide empirical evidence or extra justification to show that a quadratic approximation is valid around intermediate checkpoints in large-scale LLM pretraining?*", "methods_and_evaluation_criteria": "- **Methodology:**  \n  - The authors smartly reduce the checkpoint merging problem into a one-dimensional search by focusing on pairwise merging.\n  - They optimize the merging weight using Bayesian optimization, which is a well-suited choice for tackling expensive black-box problems.\n  - The method is evaluated on various LLM architectures and across several tasks and datasets, ensuring a comprehensive analysis.\n\n- **Evaluation Criteria:**  \n  The paper uses common metrics (like accuracy on C-Eval and CMMLU) and assesses both in-domain (IND) and out-of-domain (OOD) performance. Additionally, ablation studies (examining the effects of held-out dataset size and merging weight search space) further confirm the method’s robustness.\n\n- **Question:**  \n  - **Location:** Section 3.2 and Algorithm 1  \n  - *What is the actual runtime overhead of the GP-based Bayesian optimization compared to simpler methods like grid or random search? Could the authors include quantitative metrics or runtime comparisons to illustrate how the computational cost scales (for instance, with the number of checkpoints)?*", "theoretical_claims": "- **Proofs and Theoretical Analysis:**  \n  - The paper delivers rigorous derivations, including performance bounds from merging (see Equations (17)–(25)) and a convergence analysis in the context of gradient descent.\n  - The derivation of the PAC–Bayesian generalization bound is well-detailed (Sections D.1–D.4) and adds to the theoretical strength of the paper.\n\n- **Issues and Validation:**  \n  - Although the proofs are largely logical, some assumptions—particularly the local quadratic approximation and the boundedness of the Hessian—might need further empirical support when applied to large-scale LLMs.\n  \n- **Question:**  \n  - **Location:** Section A, Equation (15) (Quadratic Approximation)  \n  - *Can the authors provide empirical evidence or extra details to justify that a quadratic approximation holds around intermediate checkpoints in large-scale LLM pretraining? How sensitive are your conclusions if this assumption doesn’t fully hold?*", "experimental_designs_or_analyses": "- **Design Soundness:**  \n  - The experimental design is robust, featuring comparisons across various checkpoint pairs, ablation studies on held-out dataset sizes, and evaluations on models of different scales.\n  - Visual aids like Figures 1, 3, 4, and 7 effectively illustrate the trends in performance gains and the impact of key hyperparameters, such as the merging weight search space.\n\n- **Potential Improvements:**  \n  - A more explicit discussion on the computational cost of the Bayesian optimization process would be valuable.\n  \n- **Question:**  \n  - **Location:** Section 3.2 and Algorithm 1  \n  - *What is the runtime overhead introduced by the GP-based Bayesian optimization process compared to grid or random search? Could you provide quantitative metrics (such as runtime comparisons in seconds or relative speedups) to help us understand its efficiency in practice?*", "relation_to_broader_scientific_literature": "- **Integration with Previous Work:**  \n  - The paper builds on established ideas in model merging (e.g., “Model Soup” by Wortsman et al.) and extends them to the pretraining phase.\n  - It leverages Bayesian optimization—an approach widely used in Bayesian machine learning and NLP—and relates it to prior hyperparameter tuning research.\n  - The contribution fits well with current trends in improving LLM efficiency and robust training strategies.\n\n- **Missing References:**  \n  - Even though most relevant literature is cited, there might be additional recent work dealing with gradient smoothing and sharpness-aware minimization that could further support the discussion on generalization.\n  \n- **Question:**  \n  - *Have the authors considered citing and contrasting their approach with the latest studies on gradient smoothing and sharpness-aware minimization? How could these ideas enhance or further explain the theoretical underpinnings of checkpoint merging strategies?*", "essential_references_not_discussed": "Some recent studies on efficient pretraining techniques and checkpoint fusion in very large language models might be missing from the references. Including these could further situate the current contribution within the broader research context.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- **Originality:** The paper offers a fresh take by applying Bayesian optimization to checkpoint merging during LLM pretraining—an area not deeply explored before.\n- **Comprehensive Evaluation:** The wide-ranging experiments across multiple datasets and architectures convincingly substantiate the method’s effectiveness.\n- **Theoretical Rigor:** Detailed derivations provide a solid theoretical justification for improved convergence and generalization.\n\n**Weaknesses:**\n\n- **Computational Overhead:** The paper could delve deeper into understanding the added computational cost and resource usage due to the optimization step.\n- **Multi-Checkpoint Merging Discussion:** While merging two checkpoints works well, the benefits of merging more than two snapshots are less clear. A more thorough exploration of this issue would be appreciated.", "other_comments_or_suggestions": "see weaknesses above", "questions_for_authors": "1. **Computational Costs of Bayesian Optimization:**  \n   - **Location:** Section 3.2 and Algorithm 1  \n   - *What is the runtime overhead of the GP-based Bayesian optimization process compared to grid and random search? Could you include some quantitative metrics (e.g., runtime in seconds or relative speedup) to help us gauge its efficiency in practice?*\n\n2. **Generalization Across Different Architectures:**  \n   - **Location:** Section 4  \n   - *Your experiments cover Baichuan2, DeepSeek, and Pythia models. Have you noticed any architecture-specific differences in terms of merging efficiency or optimal merging weights?*\n\n3. **Multi-Checkpoint Merging:**  \n   - **Location:** Section 5.1 and Figure 7  \n   - *The results indicate that merging two checkpoints consistently provides improvements, but merging three or four checkpoints doesn’t yield significant additional gains. Can you provide more insight into why this happens, and whether there might be a way to refine the strategy to better harness the information from multiple checkpoints simultaneously?*", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4m7WQpwgLs", "reviewer": "Reviewer_sKU4", "summary": "The paper introduces a novel approach to enhancing the pretraining of large language models by leveraging intermediate checkpoint merging. The key idea is to exploit the information stored in intermediate checkpoints along the pretraining trajectory without incurring additional resource costs. The authors propose a method based on Bayesian optimization—specifically using Gaussian Processes—to determine the optimal linear blending weight for merging consecutive checkpoints. The contributions of the paper can be summarized as follows:\n\n- **Checkpoint Merging Strategy:** The paper advocates a pairwise (two-checkpoint) merging method which simplifies the high-dimensional weight search problem to a one-dimensional optimization task.\n- **Bayesian Optimization Framework:** The proposed solution utilizes Gaussian Process regression to model the relationship between the merging weight and model performance, combined with acquisition functions (such as Expected Improvement, Probability of Improvement, Upper Confidence Bound, and an adaptive GP-Hedge strategy) to efficiently search the weight space.\n- **Theoretical Analysis:** Detailed proofs are provided based on quadratic approximations, smoothness assumptions, and bounded Hessians to derive performance bounds for the merged model. Additionally, a PAC-Bayesian generalization bound is derived, indicating that the merged model can exhibit improved generalization compared to individual checkpoints.\n- **Extensive Empirical Evaluation:** Experiments are performed on multiple LLM families (Baichuan2, DeepSeek, and Pythia) across diverse benchmarks (C-Eval, CMMLU, MMLU, GSM8K, PIQA, WinoGrande, SciQ, ARC-Easy). These experiments substantiate the claims that merging adjacent checkpoints enhances accuracy and stabilizes performance across different domains.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:  Bounds on the Quadratic Approximation Error.**\n> \n\n**Our Response:**\n\n- We appreciate the request for an expanded derivation. Under the assumption that the Hessian $H(\\cdot)$ is Lipschitz continuous with constant $L_H$, consider the third-order Taylor expansion remainder term when approximating $f$ around $w_t$. For\n$\\widehat{w}_t = w_t + \\Delta,\\quad \\text{with} \\quad \\Delta = \\widehat{w}_t - w_t,$\nthe Taylor remainder $R$ can be written (via the integral form) as\n$R = \\frac{1}{2}\\int_0^1 (1-\\tau)^2 \\big[H(w_t+\\tau \\Delta) - H(w_t)\\big]\\, d\\tau \\,\\Delta^{\\otimes 2}.$\nTaking norms and using the Lipschitz property, we have:\n$\\|H(w_t+\\tau \\Delta) - H(w_t)\\| \\le L_H\\, \\tau\\|\\Delta\\|.$\nTherefore,\n$|R| \\le \\frac{1}{2}\\int_0^1 (1-\\tau)^2 L_H\\, \\tau\\, d\\tau \\,\\|\\Delta\\|^3.$\nEvaluating the integral:\n$\\int_0^1 (1-\\tau)^2 \\tau \\, d\\tau = \\frac{1}{6},$\nwe obtain the bound\n$|R| \\le \\frac{L_H}{6}\\|\\Delta\\|^3.$\n- **Practical Implications:**\n    \n    This bound is meaningful when  $\\|\\Delta\\| = \\|\\widehat{w}t - w_t\\|$ *is small—typically a valid approximation because our analysis focuses on adjacent checkpoints in the pretraining trajectory. If $\\|w_t - w{t-1}\\|$* is large (and hence $\\|\\Delta\\|$ is large, given the convex combination structure), the higher-order terms may no longer be negligible. In such cases, the quadratic approximation will incur a larger error, and our theoretical guarantees become local. Empirically, our pilot experiments demonstrate that merging distant checkpoints (which effectively produce larger $\\|\\Delta\\|$ degrades performance, confirming that our theory is most valid in the local regime.\n    \n\n---\n\n> **Q2:  Estimating the Information Gain in GP-Based Optimization ($\\gamma_T$).**\n> \n\n**Our Response:**\n\n- In one-dimensional Bayesian optimization (with a standard kernel such as the squared exponential or Matérn kernel with smoothness parameter $\\nu$, it is well established (e.g., Srinivas et al. (2010)) that\n$\\gamma_T = O(\\log T).$\nTo be precise, for a kernel $k$ on the interval $[\\alpha, 1]$ with bounded RKHS norm, one can show that the maximum information gain $\\gamma_T$ grows at most logarithmically in the number of evaluations $T$.\n\n---\n\n> **Q3:  Runtime Overhead of GP-Based Bayesian Optimization.**\n> \n\n**Our Response:**\n\n- Plz see the response in **reviewer Znoq Q2.**\n\n---\n\n> **Q4:  Influence of Kernel Choice in Gaussian Process Modeling on $\\gamma_T$ and Regret Bounds.**\n> \n\n**Our Response:**\n\n- The kernel function in GP regression encapsulates our assumptions about the smoothness and structure of the performance function $f(\\lambda_t)$. Standard choices such as the squared exponential or Matérn kernels guarantee that $\\gamma_T$ grows at most logarithmically (in one dimension), which underpins our theoretical regret bounds.\n    \n    In settings where $f(\\lambda_t)$ exhibits characteristics such as periodicity or non-stationarity, alternative kernels (e.g., periodic kernels or non-stationary kernels) could more accurately model the underlying function. This improved modeling fidelity could, in turn, lead to a smaller empirical $\\gamma_T$ and faster convergence in practice. While our theoretical development is stated in a general manner (i.e., assuming a kernel with bounded information gain), we acknowledge that kernel choice is a crucial hyperparameter. Our preliminary experiments with standard kernels (which are widely used and well understood) already yield superior performance. Future work will explore adaptive kernel selection strategies to further optimize convergence rates.\n    \n\n---\n\n> **Q5:  Clarification on the “Merging Weight” Label in Figure 3.**\n> \n\n**Our Response:**\n\n- The term “merging weight” in **Figure 3** refers to the coefficient $\\lambda_t$ in the convex combination\n$\\widetilde{\\Theta}t = \\lambda_t\\, \\Theta_t + (1-\\lambda_t)\\, \\Theta_{t-1}.$\nIt quantifies the relative contribution of the later checkpoint $\\Theta_t$ versus the earlier checkpoint $\\Theta_{t-1}$ during the merging process. The x-axis represents the value of $\\lambda_t$ uniformly sampled over the interval $[\\alpha, 1]$, where **$\\alpha$** is a lower bound ensuring that the more recent checkpoint retains a minimum contribution. We will add an explicit explanation in the revision to clarify this point.", "claims_and_evidence": "- **Major Claims:**  \n  - The intermediate checkpoint merging methodology improves model performance and convergence without incurring extra computational overhead.\n  - Bayesian optimization, when applied to optimizing the checkpoint merging weight, finds near-optimal solutions more effectively than conventional grid or random search methods.\n\n- **Supporting Evidence:**  \n  The authors present comprehensive experimental results demonstrating improved performance over various benchmarks compared to methods such as Uniform Soup, Greedy Soup, Fisher Weighted Averaging, and RegMean. The theoretical contributions are supported by rigorous proofs that derive performance bounds and convergence guarantees based on standard assumptions in optimization theory.", "methods_and_evaluation_criteria": "- **Methodological Approach:**  \n  The paper proposes merging checkpoints via a linear combination of adjacent checkpoints, thereby reducing complexity to the optimization of a single merging weight. The weight is determined by Bayesian optimization, modeled by a Gaussian Process with carefully selected acquisition functions. This framework is intended to balance exploration of the search space and exploitation of thorough performance estimates.\n\n- **Evaluation Criteria:**  \n  The methodology is evaluated using standard performance metrics (accuracy, perplexity, etc.) over a wide range of benchmarks. Although the experimental studies are extensive, the evaluation framework additionally emphasizes theoretical bounds and convergence behavior, thereby linking empirical performance with rigorous theoretical analysis.", "theoretical_claims": "- Could the authors provide a detailed derivation showing how the Lipschitz continuity of the Hessian  \n$\n\\|H(w) - H(v)\\| \\le L_H \\|w - v\\|\n$\nfor all $w, v$ leads to the above remainder bound? Additionally, please elaborate on the practical implications of this bound if $ \\|w_t - w_{t-1}\\| $ is not small.\n\n- Could the authors derive or estimate $\\gamma_T$ explicitly in this one-dimensional context, showing that\n$\n\\gamma_T = O\\big(\\log T\\big)?\n$\nMoreover, how does the dimensionality of the search space affect the overall regret bound and convergence guarantee?", "experimental_designs_or_analyses": "- **Design Soundness:**  \n  - The experimental design is robust, featuring comparisons across various checkpoint pairs, ablation studies on held-out dataset sizes, and evaluations on models of different scales.\n  - Visual aids like Figures 1, 3, 4, and 7 effectively illustrate the trends in performance gains and the impact of key hyperparameters, such as the merging weight search space.\n\n- **Potential Improvements:**  \n  - A more explicit discussion on the computational cost of the Bayesian optimization process would be valuable.\n\n- **Question:**  \n  - **Location:** Section 3.2 and Algorithm 1  \n  - *What is the runtime overhead introduced by the GP-based Bayesian optimization process compared to grid or random search? Could you provide quantitative metrics (such as runtime comparisons in seconds or relative speedups) to help us understand its efficiency in practice?*", "relation_to_broader_scientific_literature": "The manuscript situates its contributions within the existing body of work on checkpoint averaging and model merging (e.g., Uniform Soup, Greedy Soup). It also connects to a broad literature in Bayesian optimization, Gaussian Process regression, and PAC-Bayesian theory. By integrating these diverse theoretical components, the paper bridges methods in model averaging with rigorous optimization methods, offering a new perspective on efficient utilization of checkpoint information during pretraining.", "essential_references_not_discussed": "While the authors have discussed many pertinent works, additional recent studies on advanced model fusion techniques—particularly those exploring non-linear combinations or adaptive schemes for merging large-scale transformers—could provide further context. For example, works published at recent conferences on LLM optimization or papers proposing non-convex model fusion strategies might help position the contributions more clearly within evolving trends.", "other_strengths_and_weaknesses": "- **Strengths:**  \n  - The proposed methodology is innovative in exploiting intermediate checkpoints to improve performance with minimal additional cost.\n  - The combination of theoretical analysis with extensive empirical validation lends robustness to the claims.\n  - The comprehensive treatment of both convergence and generalization under a unified theoretical framework is a significant asset.\n\n- **Weaknesses:**  \n  - The theoretical derivations often rely on strong assumptions—such as local quadratic behavior, Lipschitz continuity, and bounded Hessians—which may not fully capture the complexity of modern LLM loss landscapes.\n  - Some of the derivations and notation could be clarified further to enhance readability and facilitate verification.", "other_comments_or_suggestions": "Regarding Figure 3, could the authors provide further clarification on the meaning of the x-axis label \"merging weight\" for detail explanation?", "questions_for_authors": "1. **Questions for Proof:**  \n   - see in the theoretical Claims\n\n2. **Kernel Choice in Gaussian Process Modeling:**  \n   - While the paper mentions the use of Gaussian Process regression to model the function \\(f(\\alpha_t)\\), can the authors comment on how different kernel functions might influence the information gain \\(\\gamma_T\\) and the resulting regret bound? Are there situations where a non-standard kernel might lead to better empirical convergence rates than those suggested by the theoretical analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PlTmcht0sv", "reviewer": "Reviewer_M1ML", "summary": "The paper provides an interesting Bayesian optimization based method for selecting merging weights between latest checkpoints saved during LLM pre-training. This method showed performance speed ups in empirical results and theoretical guarantees with reasonable assumptions. The initial pilot experiments were helpful to understand the motivation. Overall LLM training efficiency is an important area and this is a timely paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1:  Novel contributions compared to Previous Works.**\n> \n\n**(A) Our Focus: A Search Perspective for Pairwise Merging**\n\n- **Key Objective.** We propose to **linearly merge** two consecutive checkpoints\n$ \\widetilde{\\Theta} _t = \\lambda_t\\, \\Theta _t \\;+\\; (1 - \\lambda _t)\\, \\Theta _{t-1},\n\\quad\n\\lambda_t \\in [\\alpha, 1], $\nat each stage of large-scale pre-training, while **optimally selecting** the weight $\\lambda_t$  using a **Bayesian optimization** approach.\n- **Why Pairwise?** As we show empirically, merging adjacent checkpoints tends to preserve or enhance performance when the models are close in training steps (rather than distant). This simplification (from merging multiple checkpoints at once to only the two most recent) *drastically reduces* the dimensionality of the search to **one** parameter, $\\lambda_t$.\n- **Bayesian Optimization Rationale.** Empirically (**Figures 3--5** in our paper), the function $f(\\lambda)$ mapping the merging weight $\\lambda$ to downstream performance is **non-monotonic**. Our method:\n    1. Treats $f(\\lambda)$ as a **black-box** function (we do *not* assume it is strictly convex or linear).\n    2. Uses **Gaussian Process** (GP) modeling with an acquisition function (e.g., EI, UCB) to systematically **explore** $\\lambda\\in[\\alpha,1]$.\n    3. Finds the near-optimal or optimal $\\lambda$ in a small number of evaluations, *even if* the function is multi-modal.\n\n**(B) Distinctions from LAWA, SWA, and EMA**\n\n1. **LAWA**, **SWA**, and **EMA**:\n    - Typically rely on an **internal training procedure**—for instance, SWA requires cyclical or stepwise learning rates and continuously accumulates model parameters from multiple epochs. LAWA uses large learning rates and tail-averaging from the final phase of training. EMA uses an exponential decay factor.\n2. **Ours**:\n    - **Post-hoc approach**: We do not require mid-training gradient data, specialized cyclical learning rates, or entire logs of model state. We only need:\n        1. Several discrete checkpoints (e.g., from open-source releases or a standard training pipeline)\n        2. A small **held-out** dataset to measure performance at different $\\lambda$.\n    - **Black-box optimization**: Instead of a *fixed* schedule (as in EMA or LAWA) or a *continuous tail average* (SWA), we *actively search* for $\\lambda$ that *maximizes* the model’s performance metric. This is especially powerful for large language models whose training logs might not be fully available, and where advanced Hessian-based or continuous averaging is infeasible.\n\nHence, although both lines of work share the notion that “averaging can yield flatter minima,” our **Bayesian search-based viewpoint** and **minimal assumptions** on training steps make our method **(i) more flexible, (ii) directly data-driven for each pairwise merge, and (iii) easy to apply when only discrete checkpoints are at hand**.\n\n> **Q2:  Empirical Evidence and Comparisons**\n> \n\n**(A) Added Comparisons with LAWA, SWA, EMA**\n\nIn the revised manuscript, we include additional baselines:\n\n| Model | C-Eval (5-shot) | CMMLU (5-shot) | MMLU (5-shot) |\n| --- | --- | --- | --- |\n| Baichuan2-2200B | 54.98 | 56.29 | 51.27 |\n| Baichuan2-2420B | 54.82 | 56.78 | 53.97 |\n| Uniform Soup | 54.93 | 56.71 | 54.62 |\n| Greedy Soup | 54.64 | 56.78 | **54.82** |\n| Fisher Weighted Avg | 54.44 | 56.62 | 54.16 |\n| RegMean | 54.55 | 56.46 | 54.77 |\n| **LAWA** | 54.96 | 56.11 | 54.11 |\n| **SWA** | 56.01 | 56.66 | 54.40 |\n| **EMA** ($\\beta=0.99$) | 55.48 | 56.64 | 54.17 |\n| **Ours** | **56.23** | **56.97** | 54.56 |\n\nOur method either outperforms or closely matches these strategies, reaffirming the practical advantage.\n\n**(B) Pre-Training Perplexity**\n\nWe also conducted additional experiments on **pre-training perplexity**. As shown in the table below:\n\n| Model | CMMLU PPL | MMLU PPL |\n| --- | --- | --- |\n| Baichuan2-2200B | 5.46 | 4.87 |\n| Baichuan2-2420B | 5.46 | 4.87 |\n| Uniform Soup | 13.96 | 4.87 |\n| Greedy Soup | 5.46 | 4.87 |\n| Fisher Weighted Averaging | 14.17 | 12.45 |\n| RegMean | 5.46 | 14.54 |\n| **Ours** | **5.43** | **4.87** |\n\n> Q3: **Generalization to “Unseen” Data:**\n> \n\nWe recognize the reviewer’s concern that large LLMs might have encountered segments of certain benchmarks in their pre-training. We specifically mean “unseen” to refer to:\n\n- A dataset or domain *not* used to tune $\\lambda$. For instance, we tune on a *Chinese domain (C-Eval)* and evaluate on *English tasks (MMLU, GSM8K)*, ensuring cross-lingual or cross-domain checks.\n- Although full de-contamination is challenging, these curated benchmarks (like MMLU, GSM8K) are standard *evaluation sets* widely regarded as indicative of “test” performance beyond the direct training set.\n\n**Ref:**\n\n[1] Gaussian process optimization in the bandit setting: No regret and experimental design", "claims_and_evidence": "Some of the claims are misplaced for instance.\n\n1. Contribution a) This claim although has associated results but is not novel. For instance [1-2] specifically [1] already showed that llm pre-training can be improved through checkpoint averaging. I believe not citing key paper in one's exact area is not good practice. I fully understood that the key contribution is to find optimal weights for latest checkpoints. I believe the authors need to re-write or clarify this part.\n\nContribution b) and c) are well placed and has sufficient evidence.\n\nThe theoretical claims [line198-215] about convergence and tighter bound is all compared with regular training. Some prior works [3] have proved similar bounds and shown that LAWA [1] and [2] SWA based averaging techniques has similar bounds. I could not access the novelty or relevance of the theoretical analysis given the prior works.\n\n[1] Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. 2024. _Early weight averaging meets high learning rates for LLM pre-training._ In _First Conference on Language Modeling. (https://arxiv.org/abs/2306.03241)\n\n[2] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. Mar 2018.\n\n[3] Wang, P., Shen, L., Tao, Z., Sun, Yan Zheng, G., and Tao, D. A unified analysis for finite weight averaging. _arXiv preprint arXiv:2411.13169_, 2024b.", "methods_and_evaluation_criteria": "The authors have explained the method and evaluation criteria reasonably clearly. Here are some of my concerns\n\nThe method and set up seems to be pretty similar to LAWA [1] with some changes. Not mentioning LAWA or using it as a baseline seems not appropriate. The authors have used model souping as key baseline. In model souping multiple models are trained simultaneously with different hyper params and then souped (let's call it population averaging). I am not sure why this is a good baseline for this paper which does tail averaging again like LAWA , SWA or EMA (exponential moving average). The authors need to clarify why their baselines population averaging when there method is a variant of tail averaging.", "theoretical_claims": "I have already written  about theory claims above. I have some concerns about novelty compared to [3] (in theory). I have read the claims and skimmed over the proof I have not found anything wrong.", "experimental_designs_or_analyses": "a) The pilot studies looks super interesting. \n\nb) In Table 1 the baselines are relevant but LAWA, SWA and EMA will make these results stronger. \n\nc) The authors have used checkpoints released by various companies such as DeepSeek, Baichuan, Pythia etc. This way of evaluation is fine but [1] has some similar evaluation using pythia. But the results are still interesting and evaluation is quite robust.\n\nd) The term used in Sec 4.1 as Generalization to unseen data is a bit vague as the pre-trained models are trained on huge internet worth of data and the models might have seen many of those tasks in pre-training. I understand that you are using chinese data valset for weight selection but calling it OOD without running a de-contamination analysis seems unprincipled.\n\ne) Also an explanation or analysis of linear model connectivity among distant checkpoint could be interesting to see.", "relation_to_broader_scientific_literature": "Efficient pre-training definitely has impact. I have some concerns about very similar prior works such as [1], [2] and [3] not been discussed or treated as a baseline.", "essential_references_not_discussed": "I have some concerns about very similar prior works such as [1], [2] and [3] not been discussed or treated as a baseline.", "other_strengths_and_weaknesses": "I have already discussed previously.", "other_comments_or_suggestions": "Figure 3 seems to be placed in the wrong position as there are no reference to Figure 3.", "questions_for_authors": "1) Please list novel contributions compared to [1] for LLM pre-training.\n\n2) Do you believe the pre-train perplexity would improve with checkpoint averaging. Can you show similar results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}]

Author rebuttal:

Meta-review: This manuscript presents a new approach for enhancing the efficiency of pretraining large language models based on an intelligent checkpoint merging strategy driven by Bayesian optimization. This approach appears to work well in a series of experiments. 

The initial response to this work was somewhat mixed from the reviewers, but definitely leaning positive. There is general agreement that the methogology is sound and would be of interest to members of the ICML community. The reviewers also posed some comments and questions, which appear to have been adequately addressed by the authors during the author response period. 

Of particular note is some additional empirical results for some baselines mentioned by one reviewer; these new results should be incorporated into the manuscript.

During the discussion phase, there was general agreement from the reviewers that this work was clear, sound, and would be a compelling addition to the ICML program.
Decision: Accept (poster)