Linear Combination of Saved Checkpoints Makes Consistency and Diffusion Models Better
Authors: Enshu Liu
Junyi Zhu
Zinan Lin
Xuefei Ning
Shuaiqi Wang
Matthew B. Blaschko
Sergey Yekhanin
Shengen Yan
Guohao Dai
Huazhong Yang
Yu Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Model merging
consistency model
diffusion model
Primary area: generative models

Abstract: Diffusion Models (DM) and Consistency Models (CM) are two types of popular generative models with good generation quality on various tasks. When training DM and CM, intermediate weight checkpoints are not fully utilized and only the last converged checkpoint is used. In this work, we find proper checkpoint merging can significantly improve the training convergence and final performance. Specifically, we propose LCSC, a simple but effective and efficient method to enhance the performance of DM and CM, by combining checkpoints along the training trajectory with coefficients deduced from evolutionary search. We demonstrate the value of LCSC through two use cases: (a) Reducing training cost. With LCSC, we only need to train DM/CM with fewer number of iterations and/or lower batch sizes to obtain comparable sample quality with the fully trained model. For example, LCSC achieves considerable training speedups for CM (23$\times$ on CIFAR-10 and 15$\times$ on ImageNet-64). (b) Enhancing pre-trained models. When full training is already done, LCSC can further improve the generation quality or efficiency of the final converged models. For example,  LCSC achieves better FID using 1 number of function evaluation (NFE) than the base model with 2 NFE on consistency distillation, and decreases the NFE of DM from 15 to 9 while maintaining the generation quality. Applying LCSC to large text-to-image models, we also observe clearly enhanced generation quality.

Published as a conference paper at ICLR 2025
LINEAR COMBINATION OF SAVED CHECKPOINTS MAKES CONSISTENCY AND DIFFUSION MODELS BET- TER
Enshu Liu Junyi Zhu Zinan Lin Xuefei Ning 1:1
Shuaiqi Wang",
Matthew B. Blaschko", Sergey Yekhanin', Shengen Yan?, Huazhong Yang'.
Guohao Dañ2.6
Yu Wang!!
"Tsinghua University IInfinigence-A KKU Leuven "Camegie Mellon University Shanghai Jiao Tong University
*iirosoot Research
ABSTRACT
Diffusion Models (DM) and Consistency Models (CM) two types of pop- ular generative models with good generation quality on various tasks. When training DM and CM, intermediate weight checkpoints only the last converged checkpoint is used. this work, we find proper check- point merging can significantly improve the training convergence and final perfor- mance. Specifically, we propose LCSC, simple, effective, and efficient method to enhance the performance of DM and CM. by combining checkpoints along the training trajectory with coefficients deduced from evolutionary search. We demonstrate the value of LCSC through two use cases: (a) Reducing training cost. With LCSC, we only need to train DM/CM with fewer number of itera- tions and/or lower batch sizes to obtain comparable sample quality with the fully trained model. For example, LCSC achieves considerable training speedups for CM (23x on CIFAR-10 and 15x on ImageNet-64), (b) Enhancing pre-trained models. When full training is already done, LCSC can further improve the gen- eration quality or efficiency of the final converged models. For example, LCSC achieves better FID using number of function evaluation (NFH) than the base model with 2 NFE on consistency distillation, and decreases the NFE of DM from 15 to 9 while maintaining the generation quality. Applying LCSC to large text-to- image models, we also observe clearly enhanced generation quality.
not fully utilized and
INTRODUCTION
Diffusion Models (DMs) (Sohl-Dickstein et al. 2015 Ho et al. 2020 Song et al. 2020b) as a
generative modeling paradigm have rapidly gained widespread attention in the last several years, showing excellent performance in various tasks like image generation (Ho et al. 20200] Dhariwal & Nichol 2021 Rombach et al., 2022), video generation (Ho et al. [022] Blattann et al. (023),
and 3D generation (Poole et al. 2022 Lin et al.) 2023). DM requires an iterative denoising process in the generation process, which could be slow. Consistency Models (CMs) (Song al. (023)
proposed to handle this dilemma, which provides a better generation quality under one or few-step generation scenarios and is also broadly applied (Luo et al. 2023a Wang et al. 2023).
In this paper, we investigate the training process of DM and CM. We find that checkpoints-the model weights periodically saved during the training process-have under-exploited potential in boosting the performance of DM and CM. In particular, within the metric landscape, we observe numerous high-quality basins located near any given point along the optimization trajectory. These high-quality basins cannot be reliably reached through Stochastic Gradient Descent (SGD), includ- ing its advanced variants like Adam (Kingma & Ba 2014). However, we find that an appropriate linear combination of different checkpoints can locate these basins.
Co-first authors: Enshu Liu (1es238ma wtt.mmaais..stmtaaaaaamm www.ggui.nu..nnnt. edw.. en), Junyi Zhu (juny ai awigmmal.................aa....i com)
Co-advise
Corresponding authors: Yu Wang (yu-wang@mai 1 ttsttninggi www.tsiinnuue...... en), Xuefei
Ning (foxcooaameeggmaal.... com), and Zinan Lin (zinan) nmmi crosoft.com com).

---

Published as a conference paper at ICLR 2025
FID on Curve of LCSC CIPAR30 with
to Reration Curve of LCSC imagelett 64 with
FD-MF Curve oTUCS on CPAR-30
Vin CD- LOSC
Verill CD. 256 CD.LCSC, bs
DM
40
Voriila CD. 2045
Varilla DM
53
0
3.5
-240 sperdlu
8.50
23 13 specct
1.7 joccclp
123+ spedds
Better Pof
Trathing Benabonnxclii
Thering Beratonncdo
KFC
Figure 1: Comparison of LCSC and vanilla training. LCSC achieves more than 14x training speed up on CIFAR- 10 with Consistency Distillation (CD) and more than 1 training speed up compared to official released model on ImageNet-64 with CD. LCSC can also enhance the final converged model significantly and achieves 1.7x inference speedup for DM.
(a) Vanilla training
(b) LCSC
(c) Vanilla training
(d) LCSC
Figure 2: Comparison between the images generated by LCSC and vanilla training (on LCM-LoRA model (Luo et al. 2023a|b) b)). The prompts for the left and right images are "A car that seems to be parked illegally behind legally parked car" and "Two women waiting bench next street"" respectively. The vanilla method produces images with unrealistic front wheels (left) and an unnatural sitting posture (right), whereas LCSC produces more realistic images.
Inspired by the observation, we propose Linear Combination of Saved Checkpoints (LCSC). In particular, for a set of checkpoints saved until any point of the training process, LCSC searches for the optimal linear combination coefficients that optimize certain metrics (c.g.. FID (Heusel et al. 2017)) using an evolutionary algorithm. LCSC optimizes only in a low-dimension space and needs no pooc-propagation, thus could be faster than training. Additionally, LCSC optimize objectives whose gradients are hard to compute, under which gradient-based optimization is not applicable. We note that the widely used Exponential Moving Average (EMA) (Szegedy al., 2016) in DM and CM can be viewed as a linear combination method whose combination coefficients are determined heuristically. We will theoretically prove and empirically demonstrate that EMA coefficients are sub-optimal and are worse than LCSC. While this paper focuses on DM and CM, from a broader perspective, LCCC a general method and may be expanded to other tasks and neural networks.
We argue that LCSC is beneficial in all steps in the DM/CM production stage and can be used to: (a) Reduce training cost. The training process of DM and CM is very costly. On ImageNet-644 SOTA DMs and CMs (Dhariwal & Nichol 2021 Karras et al. 2022 Song et al. 2023 require more than 10K GPU hours on Nvidia T000. For higher resolution tasks, the resource demand soars dra- matically again: Stable Diffusion (Rombach et al.) (222) costs more than 150K GPU hours (approx. 17 years) on Nvidia A100. By applying LCSC at the end, we can train CM/DM with significantly fewer tterations smaller batch sizes and reach similar generation quality with the fully trained model, thereby reducing the computational cost of training. (b) Enhance model performance. If the full training is already done, LCSC can still be applied get a model that is better than any model in the training process. For model developers with saved checkpoints, LCSC can be directly applied. For users who can only access the final released checkpoint, they can fine-tune the checkpoint for few more iterations and apply LCSC on these checkpoints. As DM/CM provides flexible trade-off between generation quality and the number of generation steps, the enhanced model from LCSC could lead to either better generation quality or faster generation.
The reminder of this paper is organized as follows: In Sec. 2. we introduce the background and related works. In Sec. we visualize the metric landscape of DM and CM, demonstrating the potential of checkpoint merging. Additionally, we provide theoretical analyses showing that EMA is suboptimal, and that more flexible method for setting the merging coefficients is preferable. In Sec. we present our efficient method LCSC, which flexibly adjusts the merging coefficients to find the best model through evolutionary search. A schematic diagram of LCSC is shown in Fig. 3 In Sec. 5. we conduct experiments to validate the effectiveness LCSC on both DM and CM. The results show that LCSC can accelerate the training process of CM by up x3x. Moreover, LCSC
2

---

Published as a conference paper at ICLR 2025
Training Process:
Evolutionary Search:
Faster Training
Gradiett
Weight linear combination: sy +
(t), ag.
By
Perf
Mutation
Loss
Crossover
* N
1
Better Perf.
Save weights every multiple iterations.
Evaluation
Selection
Mx
1
0
ON
{araa .. FID)
Popullstion
1
10 By
Figure 3: A schematic diagram of LCSC. Given set of checkpoints from training (left), LCSC use evolutionary search to find the optimal linear combination coefficients (middle). LCSC can be applied on checkpoints from a training process with fewer training iterations or batch sizes and still gets similar performance (abbreviated training. LCSC can also enhance the final model in terms generation quality or speed (right).
"Perf."), thus reducing training cost and enabling faster
can decrease the NFE of DM from 15 to 9 while keeping the sample quality. We highlight some of the results in Fig. 1 Additionally, we demonstrate that LCSC can be applied to text-to-image
models to further improve generation quality, with some images presented in Fig. . 2 See. 6 discuss the searched patterns of coefficients to inspire further research on checkpoints merging.
we
2 BACKGROUND AND RELATED WORK
In this section, we briefly introduce the foundational concepts and related works necessary for un- derstanding this paper. A more comprehensive discussion is provided in App.
Diffusion Probabilistic Model. Let us denote the data distribution by Pitatas diffusion models (Sohl- Dickstein et al., 2015; (Song al.) 2220aa Ho et al. 2020; Nichol & Dhariwal 2021 Song et al. 2020b) learn a process that perturbs Pdato with a stochastic differential equation:
dx, (x l)dl + (l)dw,
where ()(-.-) and (-)) represent the drift and diffusion coefficients, respectively, w, denotes the standard Brownian motion, and E [0, indicates the time step. t = stands for the real data distribution. ( and (-) are designed to make sure (xx) becomes pure Gaussian noise.
To train a diffusion model, we can construct a network sø (0 t) to approximate the score function of the perturbed data distribution V Pe Px by minimizing:
E,,, Mo (ttt (tt)lssssaa)l)xxxxxx (x., ()xx,t)VVaa Vx, log //)x)))))
(2)
where X(1) represents the loss weighting and y denotes training image.
Consistency Models. During generation, diffusion models require multiple time steps, which results
in low efficiency. Consistency models (CM) are proposed to enable single-step generation (Song al. 2023), by mapping any point æ; at any given time t along a probability flow trajectory directly to the trajectory's 's initial point æ,. Training of CM can follow one of two methodologies: consistency distillation (CD) or direct consistency training (CT). In the case CD, the model fo leverages knowledge distilled from pre- iee-traieed DM é. The distillation loss can be formulated as follows:
ER-2(1.KK Ey Ens N ((. (attt)((fol (((ff((tt)))) (x+1), ) fe- (#: tk)),
(3)
where fo- refers the target model and O computed through EMA of the historical weights of 0, 4; is estimated by the pre-trained diffusion model é through one-step denoising based on #4 and d is metric such as la distance. Alternatively, in the CT case, the models fo are developed independently, without relying on any pre-trained DM:
E Betp: 1 A (lx.) (o 44112, (441,, fo- (y + 42, 4)),
where the target model fo- is set to be the as the model fe in the latest improved version of the consistency training [Song Dhariwal 2024), i.c., 0
Weight Averaging, A line of work (Ruppert 1988 Polyak & Juditsky 1992 Izmailov et al. 2018 explores the integration
running average of the weights in the context of convex optimization and stochastic gradient descent (SGD). More recently, Wortsman al. (2022) demonstrate the potential of averaging models that have been fine-tuned with various hyperparameter configurations. In early DM studies, the of Exponential Moving Average (EMA) is found to significantly enhance the
3

---

Published as a conference paper at ICLR 2025
quality of generation. This empirical strategy has since been adopted in most, if not all, subsequent research endeavors. Consequently, CM have also incorporated this technique, discovering that EMA models perform substantially better. EMA employs specific averaging form that uses the exponen- tial rate yy: 0n. - 1 + 00.... where n 1, ..... N denotes the number of training iterations, 0 represents the EMA model, and is initialized with 0u - Ou- For Large Language Models (LLMs), many works have proposed more advanced weight averaging strategies aiming at merging models fine-tuned for different downstream tasks to create new model with multiple capabilities (Ihharco
et al. 2022 Yadav et al., 2024 Jin et (0,22) Yu et al. 2023).
Unlike these approaches, our work focuses on accelerating model convergence and achieving better performance within standalone training process. Additionally, our methodology for determining merging coefficients is novel, thereby distinguishing our method from these related works.
Search-based Methods for Diffusion Models. Recently, many studies have proposed discrete op- timization dimensions for DMs and employed search methods to discover optimal solutions. This includes using search methods to optimize model schedule for DMs (Liu et al., 2023al Li ct al.) 2023 Yang et al. 2023), appropriate strategies for diffusion solvers (Liu et al. 2023b), or quantiza- tion settings for DMs (Zhao et al. 2024b a). However, these works do not involve modification to model weights and are only applicable to DMs.
3 THEORETICAL AND EMPIRICAL MOTIVATION OF LCSC
In this section, we first conduct theoretical analysis understand the effectiveness of EMA in Sec.B.1 A key insight from this analysis is that the optimal EMA rate is not fixed but depends on the number of training iterations, raising concerns about how to tune EMA effectively. We then present empirical evidence in Sec. 3.2 demonstrating that EMA is not special in its effectiveness. In fact, many linear combination results yield better performance on the trained models. Specifically, based on the empirical evidence, we provide further theoretical analysis to prove that EMA is a suboptimal method. These observations motivate the development of more flexible merging method.
3.1 THEORETICAL CONVERGENCE ANALYSIS
Analysis Framework. Previous works have established various frameworks for analyzing DMs and CMs while considering their respective objectives (Sohl-Dickstein al., 2015) Ho et al. 2020 Song et al. 2020al Karras et al.] 2022] Song et al., 2023). In this section, we follow the general optimization analysis. which represents the essential form of network optimization (Rakhlin al., 2011 Shamir& & Zhang, 2013 Harvey et al. ) Specifically, we denote f(O,) as the computed loss of the network parameters at n-th iteration in. For each training iteration n 1.....N. N, we obtain an unbiased random estimate 9n of the gradient f((.), such that E g.g.] = ff()). Addi- tionally, we denote B[llg= [|2] = a and assume that the expected square sum is upper bounded, i.e. Vn, Bilí.. [2]] < G2. We emphasize that due to the high-variance characteristic of DM and CM (see App. F.4 for a more detailed discussion), o does not diminish during training and may be signifi- cant, potentially distinguishing the training of DM and CM from other tasks. To conduct the analysis, we assume f is B-strongly convex (cf. Eq. PD). Moreover, we consider f as a non-smooth function, since natural images often exhibit features like edges and textures that lead to abrupt changes and discontinuities in their distribution. Proofs of the following analyses are provided in App. A
Based on the above analytical framework, Shamir& & Zhang (013) prove the following theorem and show that the model of the last training iteration has a convergence rate of O(ogg NN)
Theorem 3.1 (Shamir & Zhang (2013) Suppose f is 3-strongly convex, and that [[]lg,11' |] < G2 for all n I,.... N. Consider SGD with step sizes Th = 1n, then for any N 1, it holds that:
B[/(ON) - [(0')) < 17G* where O* denotes the optimal model.
The Olog( (log(NN)/) (N) convergence rate of the last-iter model is proven to be tight, as a lower bound of ((log(N) (N) is found by Rakhlin al. (2011). Next, we investigate the impact of merging the historical weights using the form: 0% = 1 Σn' a, where a = { .N. J are the merging coefficients, and A Σi Or normalizes the sum of coefficients the coefficients can be represented as a, => where 7 is the exponential rate We prove that EMA reduces significant terms in the convergence bound to O(1/N) for large N.
1. When EMA is applied,
"This formulation differs slightly from the existing EMA implementation regarding the treatment of the initial model. However, it provides mathematical convenience in the analysis. Our experimental results also show that they achieve the same performance, the discrepancy their coefficients is negligible.
4

---

Published as a conference paper at ICLR 2025
Model Near 200K Tairing
Hodel Near Training Iter
Modelx Near 200K Training lter
Models Near BOOK Training Iter
1.0
1.
6
0.3
0.0
0.3
0.0
-3.5
-0.3
EMM Near 200K Training her
EMA Near BOCK Training Iter
EMA Near 200K Trainin ther
FMA Near BOOK Training Iter
1.0
1.2
0.5
0.5
0.5
0.0
-3.5
0.5
-0.3
-10
DC-
(a) Metric Landscape DM.
(b) Metric Landscape of CM.
Figure 4: The metric landscape of DM and CM. Selected checkpoints fng 6,y. and 0ng are aligned sequentially along the training trajectory, with no < n1 < n2 denoting the progression in the number of training iterations. The origin point (0, 0) corresponds to the checkpoint Ong, while the X and Y axes quantify the differences between On Ono and One - Ono respectively. A weight located at coordinate (z. y) is formulated as ((2.)) = Pno + x00m, - 0m..)+ y(On - Bn,). Additional visualizations are provided in App. D
Theorem 3.2 Suppose f is 8-strongly convex, and that Blllgn 12 S G2 for all n. Consider SGD
with step sizes The = 3n and EMA with factor 7 E (0, 1). Then any N > [- " we have:
G²
1 (
(
2(1 2(1-~ N-1) x + +1-7).
v(7)
-
+ +
(5)
B
where [E,==>,11'>) -"
1 1-1
is constant given y and independent from N.
We observe that the first term in Thm. 3.2 decays as O(1/N), while the second term decays ponentially as )) Additionally, there is a residual of 1 - ?. Considering y is close to 1, we conclude that the significant terms (the first two terms) decay at least fast as Ol//N) and the residual term is small. We hypothesize that the improvement of these significant terms leads to an acceleration in convergence compared to the overall (logg(N) (N) rate of the last-iter model. Fur- thermore, we note that for sufficiently large N, it is preferable for to be large so that the residual approaches zero, while the enlarged factors in the first two terms can be compensated by a large N. This insight aligns with practical experience. For example, Karras et al. (2022) (024) implemented EMA rates that increase with the number of training iterations. Notably, the optimal EMA rate is difficult to predefine based on Thm. 3.2 without having the exact optimization landscape and the least upper bound. This motivates the search for EMA rates during or after the training. Moreover, in the next section, will further show that EMA is suboptimal of merging coefficients per se.
3.2 THE EFFECTIVENESS OF WEIGHT MERGING AND SUBOPTIMALITY OF EMA
To visualize the impact of linear combination of weights on the generation quality, we select 3 checkpoints :ng: Ong. Ong from the same training trajectory. Then we sweep across the 2D space spanned by these three points and assess the FID scores. The results are shown in the first row of Fig. 4 As we can see, there exists substantial opportunities to enhance the model's performance at any training phase through the linear combination of existing weight checkpoints, which proves that the effectiveness of EMA not special. Furthermore, we conduct additional experiments using EMA weights (see the seconde row of Fig. 4. Notably, we observe that linear combinations of the three EMA weights could also achieve superior performance. It is important to highlight that the linear combination of three EMA weights at various training iterations cannot be replicated by any single EMA weight according to Thm. 3.3. Consequently, this suggests that EMA is indeed a suboptimal solution, indicating possibilities for further improvement in this area.
Theorem 3.3 Assume oI...., ON are linearly independent""
| Denote .i.... 0% as the results of ".....,7 EMA using the exponential rate 7 E 1). and select three indices n1, 02, ny s.t. < mn < f2 < ng. Consider three arbitrary merging coefficients C1, C2 E (0, 1), s.f. + C2 + C3 1, we have:
A'' E 10, o + 02 + 0g0% (
(6)
This assumption likely holds, N is often substantially smaller than the number of parameters.
5

---

Published as a conference paper at ICLR 2025
Based the above insights, we conclude that the optimal weight merging coefficients difficult to predefined, and the performance of merged models can potentially be further improved with better merging coefficients. Therefore, we propose LCSC, which introduces an optimization method to search for the optimal merging coefficients.
4 METHOD
Next, we present the details of LCSC. The search problem is defined in Sec. 4.1 In Sec. 4.2 elaborate on our algorithm. Finally, we list several typical use cases of our method in Sec. 4.3
we
4.1 DEFINITION OF THE SEARCII PROBLEM
Let {1, ...., N) denote the set of training iterations. Suppose {MI, 02. ................................ nxk } is subsequence selected from (1.2..... N}. Define θ as the set of checkpoints saved at these specific training
iterations: O = {0n, Onz: Ons: Onx). Given O we aim to find group of coefficients a
{(r1, 02, 03. ... .ak). j. that achieves the best utility when linearly combine all checkpoints:
K
arg min ,1 a1,0g,00, 0rg, where F(0) denotes an evaluation function that measures the generation quality of 0 regarding a specific metric, indicating higher quality with smaller value. The constraint > a; = reduces the dimension of the search space by 1. However, we find it enables a more effective exploration for the search algorithm. Additionally, since the basin area in Fig. extends beyond the convex hull of the model weights, we allow a < 0, distinguishing LCSC from existing weight averaging methods.
O,, + azgnn + a.30ng + +.... + 0xx0nx)
s.t. _a=
=
(7)
ER
[=1
The trained model can also be a Low Rank Adapter (LoRA) (Hu et al. 2021). In this casc, each checkpoint a product of two low-rank matrices; On, Bn, An. The weighted sum of all check- points by the coefficients is: 01 Ba, Ang 0:2 Bng Anz 0zg Brg Ang
og B Ang-
4.2 EVOLUTIONARY SEARCH
As discussed in Sec. 3, the optimal coefficients are difficult predefine. To solve the problem in Sec. 4.1 we propose to use evolutionary search (Real et al. 2019 Liu et al. 2023a Li et al. 2023).
Our detailed algorithm is provided in Alg. I and illustration is given in Fig. 3 Given set of checkpoints O, we can apply LCSC at any training iteration n along the training trajectory. The key points of our algorithm are summarized subtract the first checkpoint from each subsequent checkpoint and search for the coefficients of their difference. (b) We initialize the population using EMA coefficients of several different rates. (c) We then conduct an evolutionary search which views group of coefficients as an individual. In each search iteration, only the oop-prffrrming individuals are selected as parents to reproduce the next generation through crossover and mutation. Specifically, during crossover, we randomly mix the coefficients of the two parents or select one set entirely. For mutation, we introduce random Gaussian noise to each coefficient. This stochastic element ensures that beneficial adaptations conserved and advanced to the subsequent generation, whereas detrimental modifications are discarded. (d) We repeat this reproduction process for predetermined number of times within each search iteration and update the whole population with all newly generated individuals. (e) Upon the completion of the search process, we choose the best individual the output of our search algorithm.
follows: (a) For the stability during search, we
Since evolutionary search only performs model inference and is thus gradient-free, LCSC achieves significant savings in GPU memory usage (applicable both DM and CM) and computational time Further-
(particularly for CM, given that the inference process for DM is still essource-intensiin more, LCSC has the unique advantage of optimizing models directly for non-differentiable metrics. The applicability of LCSC could potentially extends to various other tasks and models as well.
4.3 USE CASES
Finally, we delve into the use cases of LCSC. (a) Decrease Training Cost. Through our investiga- tion, we have discerned that the training process for DM and CM can be categorized into two distinct phases, with the second phase taking the majority of training iterations but converging slowly (fur- ther discussion is deferred to App. E.2) Our method can be employed at the onset of the second phase and yield performance on par with or even surpasses the final converged model. Additionally, SOTA DMs and CMs necessitate training with large batch sizes, which places substantial demand on computational resources (Karras et al. 2022; Dhariwal & Nichol 2021 I. [Sog et al., 2023). De- creasing the batch size often leads to worse convergence speed. LCSC be used for low batch

---

Published as a conference paper at ICLR 2025
size training and achieve equivalent performance to models trained with full batch size. These ap- plications aims to expedite the training process. (b) Enhance Converged Models. LCSC can be applied to the checkpoints saved during the final stage of training to refine the converged model. Ir can enhance the generation quality under the same NFE, or maintain the generation quality with fewer NFE, thereby reducing the inference cost during deployment. Additionally, for users who only have access to released pre- pre-trained models, we suggest fine- fine-tuning the model for a few iterations and saving the intermediate checkpoints. Then LCSC can be used enhance the released model.
EXPERIMENTS
In Sec. 5.1 we introduce the experimental settings. In Sec. 5.2 and 5.3 we demonstrate results for the two use cases. In Sec. 5.4 we demonstrate the potential of merging LoRA checkpoints. We further apply our method on text-to-imaae task in (.5) Finally, we discuss the generalization ability of LCSC in Sec. 5.6
We ablate several important hyper-parameee in our workflow in
App. E.4
5.1
EXPERIMENTAL SETUP
We follow previous work configure the training process. Details of the training and search con- figurations are provided in App. E.I
Evaluation. We choose the most commonly used metric, FID (Heusel et al. 2017), as the search objective. To evaluate the combination coefficients during the evolutionary search, we sample 5K images for ImageNet-64 with DM and LSUN datasets, while 10K for other settings, using fixed group of initial noise. For the final model evaluation, we generate 50K samples using a different group of initial noises. To demonstrate that LCSC searches based FID but achieves overall improvement, we also report PickScore (Kirstain et al.] (023), ImageReward (Xu et al., 2023)) and the winning rates based on these two metrics for the text-to-image task, and IS (Salimans al., 2016), precision, and recall KKykäääniiemi et al. 2019) for other cases. In |cc. we further report FCD (the variant of FID using CLIP features instead of Inception features) and KID (Binkowski et al. 2018) to show that LCSC also brings improvements in other feature spaces or metrics.
Baselines. Since EMA is the default setting in almost all works of DM and CM, we report the per- formance of the EMA weights for full-model training using the rate reported by official papers (Song) al. 2023 2020a| Nichol Dhariwal 202T) as our main baselines. For CM models on ImageNet- 64 and LSUN datasets, we additionally download the official models trained with full batch size and test their performance for a complete comparison. We further conduct grid search of EMA rate for the final model as stronger baseline, denoted as EMA*, to prove the sub-optimality of EMA and that LCSC is a more effective method. More experimental details are provided App.
5.2 RESULTS ON REDUCING THE TRAINING COST
We conduct a series of experiments on CD and CT to show that LCSC can significantly reduce their training costs. The efficiency of LCSC has considered both training and search cost the CPU and GPU, with detailed information available in App. E.3 Generated images are visualized in App. H
LCSC can be applied illustrated Tab. | applying LCSC to the CD model with only 50K training iterations achieves a better FID than the final model trained with 800K iterations on CIFAR- 10 (3.10 vs. 3.66), which accelerates the training by approximately 14x. For CT, we only achieve a converged FID of 9.87 with vanilla training and are unable fully reproduce the result of FID=8.77 70 reported by [Song al. (203). However, by applying our method at 400K training iterations, we can achieve a similar FID the reported one from 800K training iterations, achieving around 1.9x speedup. On ImageNet- 64 (Tab. (2), we decrease the batch size from 2048 to 256 due to limited resources. For CD, LCSC achieves better performance (5.51 vs. 7.30 in FID) than the final converged model with only half of the training iterations (300K vs. 600K). For CT. LCSC also significantly reduces the final converged FID (10.5 vs. 15.6) with fewer training iterations (600K vs. 1000K).
early training stage reduce the number of training iterations. As
LCSC can also handle smaller training batch sizes to achieve higher speedups. As illustrated in Tab. [ LCSC with a batch size of 128 outperforms the final converged model with a batch size of 512 for both CD and CT (3.21 vs. 3.66 with CD and 8.54 vs. 9.87 with CT), achieving overall speedups of 23 x and 7x, respectively. On ImageNet-64, we test the official models of SSong et al. 2023). which were trained with 2048 batch size for both CD and CT. With the assistance of LCSCC.
7

---

Published as a conference paper at ICLR 2025
Table 1: Generation quality of CMs on CIFAR-10. The training speedup compared against the
standard training with 800K iterations and 512 batch size. Our results that beat match the standard training (the "released" model for FID & IS, our reproduced results for Prec. & Rec.) are underlined.
Model
Method
Training
Batch Size
NFE
FID(L)
IS(1)
Prec.(t)
Rec.(t)
Speed(1)
200K
512
4.08
9.18
0.68
0.56
800K
512
3.66
9.35
0.68
0.57
850K
512
3.65
9.32
0.68
0.57
EMA
850K
512
2.89
9.55
0.69
0.58
(relessd))
800K
512
3.55
9.48
(release))
800K
512
2.93
9.75
800K
512
3.51
9.37
0.68
0.57
50K
3.10
0.66
0.58
~44
800K
512
0.67
0.60
LCSC
800+-40K
512
2.50
9.70
0.68
0.59
100K
128
3.21
9.48
0.66
0.58
400K
512
12.1
8.52
0.67
0.43
800K
512
9.87
8.81
0.69
0.42
800K
512
8.70
8.49
800K
512
9.70
8.81
0.69
0.42
400K
512
8.79
0.67
0.47
~1,9x
LOSC
800++40K
512
7.05
9.01
0.70
0.45
450K
128
8.54
8.66
0.69
0.44
~7
Table 2: Generation quality of consistency models on ImageNet-64. For CD, the speedup is com- pared against the standard training with 600K iterations and 2048 batch size. For CT, the speedup is compared against the standard training with 800K iterations and 2048 batch size. Our results that beat the standard training ("released") are underlined.
Model Method
Training Iter
Batch Size
NFE
FID(J)
IS(1)
Prec.(t)
Rec.(f)
Speed(1))
300K
256
7.70
37.0
0.67
0.62
600K
256
7.30
37.2
0.67
0.62
EMA
650K
256
7.17
37.7
0.67
0.62
CD
(relensed)
600K
2048
6.31
39.5
0.68
0.63
EMA
600K
256
7.17
37.7
0.67
0.62
300K
256
5.51
39.8
0.68
0.62
15x
LCSC
600+20K
256
5.07
42.5
0.69
0.62
~766
600K
256
16.6
30.6
0.62
0.54
EMA
800K
256
15.8
31.1
0.64
0.55
1000K
256
15.6
31.2
0.64
0.55
(reloused)
800K
2048
13.1
29.2
0.70
0.47
1000K
256
15.6
31.2
0.64
0.55
600K
256
10.5
36.8
0.66
0.56
11.4x
LCSC
800K
256
9.02
38.8
0.68
0.55
~7.x
models trained with 256 batch size consistently outperform the official models (5.51, 5.07 with CD and 9.02, 10.5 vs. 13.1 with CT), achieving overall acceleration ratios to 15x.
6.31
5.3 RESULTS ON ENHANCING PRE-TRAINED MODELS
Our experiments on both DM and CM show that LCSC can significantly boost the performance of final converged models and reduce the required sampling steps.
To simulate scenarios where users have access only to pre-trained models, we fine-tune the final converged model for few iterations and then apply LCSC. As illustrated in the 800+40K LCSC row with CD and CT in Tab. | the 600+20K LCSC row with CD in Tab, Tab.2land the 1000+20K LCSC on both datasets Tab. 3 this approach significantly enhances the performance of the already converged model. For model developers, they can directly utilize the checkpoints close to the final iteration, as demonstrated in the 800K LCSC row with CD Tab. and 800K LCSSC row with CT in Tab. LCSC leads to notable improvements in all these scenarios. Notably, CIFAR- 10, LCSC even outperforms the 800K and 850K models that use 2-step sampling, achieving better performance with 1-step sampling (2.44, 2.50 FID vs. 2.93, 2.89 FID), thereby doubling the inference speed. More experiments with Stable Diffusion checkpoints can be found at App. E.8
Same phenomenon has also been observed for DM, as shown in Tab. 4 applying LCSC at the final stage of training can obtain a model that performs significantly better than the final model of EMA. Moreover, observe that the model found by LCSC can match the performance of the best EMA
8

---

Published as a conference paper at ICLR 2025
Table 3: Generation quality of models on LSUN-bedroom and LSUN-cat with consistency training (CT), We get the checkpoints by fine-tuning the released official model.
Dataset
Method
Training Iter
Batch Size
NFE
FID(C)
Prec.(1)
Rec.(t)
1000K (relcased)
2048
1
20.8
0.53
0.46
LSUN-Cat
EMA
1000+20K (fine-tuned)
256
1
21.0
0.58
0.44
1000+20K
256
17.8
0.63
0.48
(000((releuscd)
2048
1
16.1
0.60
0.17
LSUN-Bedroom
EMA
(f00-t0K(())etunnel)
256
16.0
0.60
0.17
1000+20K
256
13.5
0.59
0.37
Dataset
Method
Training Iter.
NFE
FID(J)
IS(1)
Prec.(1)
Rec.(t)
150K
15
6.28
8.74
0.61
0.59
EMA
800K
15
4.16
9.37
0.64
0.60
EMA
800K
3.96
9.50
0.64
0.60
Table Geneeatton
CIEARIO
150K
4.76
8.97
0.61
0.59
quality of diffusion
LOSC
800K
15
3.18
9.59
0.64
0.61
models. Our results that beat the standard training are underlined.
800K
3.97
9.50
0.63
0.60
150K
15
22.3
15.0
0.55
0.45
EMA
500K
15
19.8
16.9
0.58
0.59
EMA
500K
18.1
17.3
0.59
0.59
ImageNet
150K
19.1
15.3
0.56
0,56
LCSC
500K
15.3
17.6
0.59
0.59
500K
12
17.2
17.2
0.57
0.59
Table 5: Single-step generation quality of LoRA models.
Dataset
Model
Method
Training Iter
Batch Size
FID())
IS (f)
Prec.(")
Rec.(1)
EMA
600K
256
7.30
37.21
0.67
0.62
LoRA
600+20K (fine-tuned with LoRA)
256
6.79
37.74
0.67
600+20K
4.21
43.22
ImageNet
800K
256
15.75
31.08
0.64
0.55
LoRA
800+20K tfine-tuned with LoRA)
256
15.14
31.59
0.64
0.56
LCSC
800+20K
256
4.90
45.33
0.67
0.65
EMA
1000K
2048
16.10
0.60
0.17
L.UUN-oodmom
LCSC
1000+0KK
256
14,49
0.60
0.20
Table 6: Text-to-image generation quality LCM LoRA. PKS and IR stand for PickScore and
ImageReward. WR stands for winning rate among all generated images compared to baselines.
Method
Training
Batch Size
Seurch NTE
Eval NFE
FIX()
PKS(1)
WRGPKS(1)
IR(
WR@IR(1)
CLIP
Vanilla training
12
4
32.52
0.46
34%
-2.20
35%
26.02
LCSC
6K
12
4
4
28.30
0.54
66%
-2.99
65%
26.39
Vanilla training
12
0.46
33%
23%
LCSC
12
2
2
30.39
0.54
67%
-2.20
77%
26.01
Vanilla training LCSC
6K
12
2
43.32
0.47
34%
-2.22
25%
25.16
12
0.53
75%
model with fewer NFE during inference. Specifically, method requiring NFE=9 is on par with EMA* using NFE=15 on CIFAR-10. Similarly, our method with NFE=12 is competitive with EMA* utilizing NFE=15 ImageNet-64. This also highlights the potential for inference speedup.
5.4 RESULTS WITH LORA CHECKPOINTS
As discussed in Sec. 4.1 we fine-tune the model with LoRA and linearly combine the LoRA checkpoints, which significantly reduces the demand for memory and storage compared to full- model LCSC. Results are reported in Tab. [5] Fine-tuning with LoRA checkpoints achieves per- formance comparable to full-model fine-tuning with LCSC. On ImageNet-64 dataset with CT, we surprisingly find that applying LCSC LoRA checkpoints outperforms the full model by substan- tial margin, highlighting the effectiveness of combining LCSC with LoRA.
5.5 RESULTS ON TEXT-TO-IMAGE TASK
To further validate the practicality of LCSC, we apply it to the training process of LCM-LoRA (Luo et al. 2023a b). We train the model on CC12M dataset and use 1k extra image-text pair in this dataset to conduct search. We apply 4-step sampling and 2-step sampling and observe that the search results from 4-step sampling can generalize to 2-step evaluation. We test the search result using 10k data in MS-COCO dataset with FID, PickScore, and ImageReward. Since PickScore and ImageReward are both image-wise metrics, we also report the winning rate of LCSC compared to baseline for these two metrics. More details of the setting can found in App. E.1.3 Results are shown at Tab, 6 several examples are visualized in Fig. and more provided in Fig. 16 LCSC can significantly

---

Published as a conference paper at ICLR 2025
Table 7: Evaluation results using FID, FCD on the training dataset and KID the test dataset. See Tab. T8 for test set FID and Tab. 17 for results on CT.
CIFAR-10.
Results on ImageNet-64.
Method
Training Iter
Batch Size
FID(J)
FCD())
KID(.)
Method
Training Iter
Batch Size
FID(J)
PCD())
KID(J)
CD
800k
512
3,66
18.0
1.38c3
CD
300k
256
7.70
30.0
3,980-3
S00k
1.39--
CD
600K
256
7.30
29.1
3.65e-3
LCSC
100k
128
3.21
23.4
7.88c-4
CD
600k
2048
6.31
25.5
3,370-3
LCSC
250k
512
2.66
14.3
4.43c-1
LCSC
300k
256
5.51
23.9
3.14
LCSC
800k
2.44
4.32e-4
LCSC
800+40k
256
5.07
23.2
1.97c-3
LCSC
800+H0K
512
2.50
14,0
4.11e-4
Results un CIFAR-10 with DM
Resuls on CIFAR-10 with CD
Results on CIFAR-10 with CT
0.7
Starched Weights
EMA 0.9999
Searthed wicights EMA 0.9399
Searched Weights
EMA 0,9959
0.0
-0.
-1.0
-0.2
-0.
76 76 70
75 77 70 Training Reerationx10KI
37.8 37.5 360 305 3E.0 355 42.0 Training Iteration
Training [ttaationxx1KK)
Figure 5: Visualization of weight combination coefficients obtained using LCSC and EMA. boost the FID. For human evaluation based metrics, LCSC achieves both better average value and higher winning rate, indicating the potential of LCSC to be used in more complicated tasks.
5.6 DISCUSSION ON THE GENERALIZATION OF LCSC
Even if LCSC searches based on evaluation metries (e.g., FID), we show here that LCSC generalizes across metrics and data.
Metric Generalization. In Sec. 5.2 5.5 we have evaluated the search results across various
metrics besides FID, including IS, Precision, Recall, PickScore, and ImageReward. Additionally, we further evaluate the search results on the test set using KID (Binkowski et al. 2018 and FCD (i.e., replacing Inception Net with CLIP as the feature extractor for FID); see Tab. 7 and Tab. 17 These metrics differ from FID various aspects, including feature extractors (FCD, PickScore, IwageReward), calculation formulas (IS, Precision, Recall, KID, PickScore, and Image application scenarios (PickScore and ImageReward for human- evaluation). We observe that most metrics consistently improved, demonstrating the generalizability of LCSC across metrics.
Data Generalization. (a) Generalization of initial noise. During the search process, a fixed group of initial noise is used. When performing evaluation, search result of LCSC generalizes to different initial noise. (b) Generalization of data. Since FID computed using the training test set to validate that the search results of LCSC generalize across data; see Tabs. 7 and T8 Addi- tionally, for text-to-image task (see Sec. (.5)). LCSC was searched based on CC12M but evaluated on the test dataset of MS-COCO, demonstrating generalization to a closely related data distribution.
different set of noise. Therefore, the
the ground truth, we calculate additional metrics based on the
6 DISCUSSION AND CONCLUSION
Analysis of Search Patterns. Since our method searches for the optimal coefficients, it would be interesting to check if the search pattern aligns with previous methods using fixed forms, such as EMA, and to derive insights that could inspire further research checkpoint merging. We visualize several searched combination coefficients in Fig. (a) First, we observe that earlier checkpoints can also be important, as some of them have large coefficients. (b) Moreover, we find that LCSC tend to assign large coefficients to a small subset of weights. whereas the coefficients for the majority of weights are nearly zero. Further investigation suggests smaller and homogeneous solution also exists but may be difficult to be found by LCSC (seeApp. F). (c) Finally, the presence of multiple significant negative coefficients highlights that certain weights can act as critical negative examples. This finding implies traditional weight-averaging methods, such as EMA, are suboptimal. Since they commonly confine the resulting model within the convex hull of all weights, which excludes the discovered solutions. More search patterns and other insights are provided in App. F
In this work, we investigate linearly combining saved checkpoints during training to achieve bet- ter performance for DM and CM. We demonstrate the common benefits of checkpoint merging and provide a theoretical analysis to clarify that the current standard merging method is suboptimal, emphasizing the need for flexible merging coefficients. We then propose using an evolutionary algo- rithm to search for the optimal coefficients, which runs efficiently. Through extensive experiments, demonstrate two uses of our method: reducing training costs and enhancing generation quality.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by National Natural Science Foundation of China (No. 62325405, 62104128, U19B2019, U21B2031, 61832007, 62204164), Flemish Government (AI Research Pro- gram) and the Research Foundation Flanders (FWO) through project number GOG292IN, Ts- inghua EE Xilinx AI Research Fund, and Beijing National Research Center for Information Science and Technology (BNRist). We thank the anonymous reviewers for their valuable feedback and sug- gestions. We thank Yiran Shi for his help with experiments and the support from Infinigence-Al.
REFERENCES
David Berthelot, Arnaud Autef. Jierui Lin, Dian Ang Yap. Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303.04248 2023.
Mikolaj Binkowski,, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 01 401, 2018.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models large datasets, arXiv preprint arXiv:2311.15127, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wil- son. Loss surfaces, mode connectivity, and fast ensembling of dnns. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL
nwwrips. cwtppprrrcppppppppp.ppppepp files/paper/2018/ pdf
File
Hyojun Go, Yunsung Lee, Seunghyun Lee, Shinhyeok Oh, Hyeongdon Moon, and Seungtaek Choi. Addressing negative transfer in diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7441-7451, 2023.
Nicholas JA Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non- smooth stochastic gradient descent. In Conference on Learning Theory, pp. 1579-1613, PMLR, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter: Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint 22X 2022.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2Xiv:2/06.09666 2021.
11

---

Published as a conference paper at ICLR 2025
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-70) 2005.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi. and Ali Farhadi. Editing models with task arithmetic. arXiv preprint S2KV::2212.0408,, 2022.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. In Ricardo Silva, Amir Globerson, and Amir Globerson (eds.), 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp- 876 885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09844, 2022.
Alexia JolicoeurMMattineau Ke Li, Rémi Piché-Taillefer Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint A.iv:2105,140300 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 25:26775-26577, 2022.
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 2014.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Tuomas Kynkääänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-free optimization of time steps and architec- tures for automated diffusion model acceleration. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 7105-7114, 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 300-309, 2023.
Enshu Liu, Xuefei Ning, Zinan Lin, Huazhong Yang, and Yu Wang. Oms-dpm: Optimizing the model schedule for diffusion probabilistic models. arXiv preprint wxXiv:2306.08866 2023a.
Enshu Liu, Xuefei Ning, Huazhong Yang, and Yu Wang. A unified sampling framework for solver searching of diffusion probabilistic models. arXiv preprint arXiv:2312.07243 2023b.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022. URL https: //openreviess www./ttttoote d=P kwVd2yRkY
12

---

Published as a conference paper at ICLR 2025
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.00922 000220 2022.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04377 2023a.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint X 1223311055 1.0556,, 2023b.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171.
PMLR, 18-24 Jul 2021.
Xuefei Ning. Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-base based nas. In European Conference on Com- puter Vision, pp. 189-204. Springer, 2020.
B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(():838-855 1992.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion, arXiv preprint arXiv:2209. 14988, 2022.
Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5644 2011.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 16684-16993 2022.
David Ruppert. Efficient estimations from slowly convergent robbins-monry process. Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Conver- gence results and optimal averaging schemes. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume Proceed- ings of Machine Learning Research, pp. 71-79, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, October 2020a. URL
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. In The Twelfth International Conference on Learning Representations, 2024.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011 13446, 2020b.
13

---

Published as a conference paper at ICLR 2025
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zhigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolem: Video latent consistency model. arXiv preprint arXiv:23 12.09109, 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre. Rebecca Roelofs, Raphael Gontijo-Lpees, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965- 23998. PMLR, 17-23 Jul 2022.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Shuai Yang. Yukang Chen, Luozhou Wang, Shu Liu, and Yingcong Chen. Denoising diffusion step-aware models. arXiv preprint arXiv:2310.03337 2023.
Le Yu, Bowen Yu. Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. arXiv preprint arXiv:2311.0309) 2023.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
IEEE Computer Society, 2018
Tianchen Zhao, Tongcheng Fang, Enshu Liu, Rui Wan, Widyadewi Soedarmadji, Shiyao Li, Zinan Lin, Guohao Dai, Shengen Yan, Huazhong Yang, al. Vidit-q: Efficient and accurate quantiza- tion of diffusion transformers for image and video generation. arXiv preprint arXiv:2406. 02540, 2024a.
Tianchen Zhao, Xuefei Ning. Tongcheng Fang, Enshu Liu, Guyue Huang, Zinan Lin, Shengen Yan, Guohao Dai, and Yu Wang- Mixdq: Memory-efficiet few-step text-to-image diffusion mod- els with metric-decoupled mixed precision quantization. In European Conference on Computer Vision, pp. 285-302. Springer, 2024b.
A PROOF OF THEORETICAL ANALYSIS
Lemma A.1 Suppose f is B-strongly convex, and that [|] < G2 for all I.
Consider SGD with step sizes Tn = 1/3n, then for any > 1, holds that:
2G
E1|00* - 1-
(8)
3nn
Proof of Lem. A.I basically follows Rakhlin et al. 2011), while we improve the upper bound with factor of 2.
By the strong convexity of f, we have:
3 0 0
(g.. O, 0²) 2 f(On) f(0")
(9)
2
3 0,, 0 1-
-(0m)-((0') - 22
(10)
14

---

Published as a conference paper at ICLR 2025
Based on the above inequalities, derive:
0 8* 0= | Mn.. 1*
(11)
O, 0-1 o B1lxx 112] 21, E((O,, 0", 0-, 99))
(12)
O 0- |-] + G² 22, 00 90))
(13)
8
3
0 11' + 0² 2// (S s(0))
0n
0- 13))
(14)
2
L0
||0,,. 0- 1²= n G 21/ (1 0-1
(15)
Plugging in 7/ 1/,, obtain:
- 001 2 >E[10 1- +
G²
(16)
pnn
When n = 1, we have:
G
G
E| E| 02 1'' Σ B| --1 +
g2 S
(17)
B²
Since Bi|022 00113]] 2 0, we can also obtain:
G²
E|
0- 1121 S 3z
(18)
Therefore, when n 1.2, Lem. A.l satisfied. For n 2 2, we can further prove it using induction agreement:
2 0-113] S (1
G²
[I 0-
(19)
n
Bpn2
s (1 2 2G2
G
+
(20)
n B2n
(2n G)
(21)
32n2
2G2
(22)
(2
A.1 PROOF OF THM. 3.2
Rearrange Eq. (2):
O 0" gn 1 (E] 0-1 11[10 - 00(13)) + Ta
Bllggn 12].
(23)
2//m
Plugging in Eq. 9 and nn = 1/3nn, we obtain:
8
O(,( - f(0)) )] 5 (nn - 1) Ol - 0-113] - nE[1 - -00133)
G².
(24)
2
2,8n
Summing over N iterations using coefficients a::
B( - s(0)))
(25)
Σ
N
On - 1I - 1- nEE[ 0 - 11)) +
G2
N
Og.
(26)
2A
2,3A
n=l
n=!
G2
N
Σ (an an- (n 1) E1]IO, -
071131
+
2A
233
n=
n=l
(27)
9
G2
N
Σ
(an an-1) (n (1-1) 11+
(28)
2A
2,3A
15

---

Published as a conference paper at ICLR 2025
Plugging in Lem. A.1
we have:
BlS(OR) - 1(0')
5
G
an an-1) (n 1)
G
Σ
an
(29)
3A
23A
T
n=
G
N
G
N
Og
aan an
+
(30)
3A
23A
2=2
n=1
Plugging in on = 1-
have
G
N
G2
N
N-n
E[ - f(0')] <
N-
7
(N-tt))+
+
(31)
BA
23A
T1
2=2
==1
G3
N
G²
N
N-n
(32)
BA
23A
n
7=2
R=1
G2(1 - 7) 1 1-1 N-2
G
(33)
BA
1 1-7
23A
7
GP(1
1 1 N-2
G2
7)
1
(34)
N-
23(1
N-1)
n
G
7)
(1
N
1
(35)
3
2 - N-1)
Next, we focus on deriving an upper bound for the second term in Eq. 35 since there is no simple closed-form expression for it. We notice that y decays faster than n grows. Therefore it is more important to evaluate y when n approaches N.
We will slightly abuse the notation x to denote a positive real number. First, we note that:
1 d_ = -1="x) + xn In
d 75x
(36)
1/x + In
(37)
x
If x 2 -in 1_ we have 1/x & In 1/ V7. Plug this into Eq. 37
In
1 In
i In V
(38)
x
In VT
(39)
7
Integrating both sides from x = [ 19 to r
N 1, we obtain:
N+1
d
(40)
N +1)
7 l
N
/7
dx.
(41)
in
m
16

---

Published as a conference paper at ICLR 2025
Since the derivative in Eq. (39) is always positive, the function 7F when x 2 - In /r. In such case, the lower Riemann sums underestimate the integral, we have:
is monotonically increasing
N
N+1
1 dr
(42)
7
7a
==
(43)
In
IN +
In
1
1
(44)
7N + 1) /7
Plugging Eq.
into Eq. 35 we have:
1-1
G2 (1-7)
G2 - 7)
N
(() f(0+))
23(1 - 7 N 11-) (
Σ
(45)
75
j=1
G2 7)
(2 -7)
23(1 7 N-11 ;M(
1-1
(46)
NN + 1) In V 7 m
j=1
G2(1-))
G
7
+ +>N0(>))
(47)
2 N -1)
?(N + 1) In 1/ V7
G² - -
G2
?(N + 1) + M77))))
(48)
B
2 N
G2
2(1 y N- 7M + 1 +1-7), 7).
v(7)
+
(49)
711 N- NNN +1)
where v(7) =
Ex=1
To derive Eq. (48). we first identify that (1
7) In 1/ is monotonically increasing for 7 E (0, 1). We then compute the limit as y + using L'Hôpital's rule.
A.2 PROOF OF THM.
We prove Thm. 3.3 using proof by contradiction.
Assume 3y' (0. 1), s.t. 0107, + +20%% + c300% (0 0... Denote n' E {1...N} such that:
0 t20%, + cs0 = 0"..
(50)
Since ...... 0% are the EMA of ......, ., ON. which are linearly independent, and 0% an EMA of ....... 0x., we can derive: n* = n2.
Substituting the EMA models with the original models, we have:
c,07 + +c20% c307, = + "2 + #37 -1)0 +...
(1 7) (cr c27 "2- m + ~2 "2-11 Ong +...
+ (1 ()2 + 037 No -2)= 0nn +... (1 - (1-7)c30 ns.
(51)
07 =1 - +... + (1 / m m On, + (1 7/) )/ ms 0 n2 + (1 - (1-71)0. 0m
+...
(52)
Note that we apply the practical implementation of EMA for Thm. 3.3 which is different from
Thm. 3.2 However, their discrepancy is negligible. Since 0Y,... ..... Ny are linearly independent,
17

---

Eq. 60 contradicts with the assumption y, 1). Published as a conference paper at ICLR 2025
coefficients of all models 0 ...... Ons in Eqs. (51) and (2)
should be aliigedd with each other. Based
on the last term in Eqs. (11) and (52). we have:
(1 7) = 2
(53)
Case n2 ng 1 for the index 72 - 1:
First, we consider the case n2 < n2 1, we have the following equations
(1- 00nn- ns-
(54)
7)703
7/1
(55)
733
53
7)0r7
(56)
(57)
(1 7/003
(58)
7(1 03) 1 03
(59)
7 = 1.
(60)
Case 2: n2 n2 1 and 71 < 122 1 Next, we consider the case n2 1i3 and ni < f22 1.
We have the following equation for the index ne2:
7'n''n'0, Ong (1 7) r33 r
+ (1 - 7)02 1
(61)
Additionally, we have the equations below for the index 122 1:
(1 - ,0
- 7) 7203 + +(1 (1- (1-7)700 702
(62)
(11 + (1 7)
7) 703 + (1 (117))0)) 02)
(63)
(64)
57 60
(65)
Eq. 665 contradicts with the assumption 7, 7, 1).
Case 3: n2 = n2 - 1 and n1 = n2 n Finally, we consider the case n2 = n1 P3--1 and n1 = 722-1 1.
We have the following equation for the index n1!
(1->)>>² (1 - ? + (1-7) 702 + - 7)
(66)
Additionally, we have the equations below for the index
1: -
(1 (1-1h//3 7) + (1 - -) + (1 -1- 7)701 (67)
((1-7) (1- r02 + (1- 7
(1-7)) 7 + (1- 702 (1 )))) γ
(68)
(69)
57
(70)
Eq. 70
contradicts with the condition y, y E (0, 1).
Since the condition 7.76 E (0, 1) always leads to contradiction, prove that 2n' (0, 1), s.t.
c1 + +2 + c E {0) ,...,,), 0%
B EXTENDED BACKGROUND AND RELATED WORK
B. DIFFUSION PROBABILISTIC MODEL
Let us denote the data distribution by Pdata and consider a diffusion process that perturbs Paato with a stochastic differential equation (SDE) (Song et al. 2020b);
dx, 1x(x,, f)dt (I)dwr,
(71)
18

---

Published as a conference paper at ICLR 2025
where (e(-.-) and o.)) represent the drift and diffusion coefficients, respectively, wy denotes the standard Brownian motion, and E [0, T'] indicates the time step. t = stands for the real data distribution. ( and (-) are designed to make sure (x becomes pure Gaussian noise.
Diffusion models (DM)
et al. 2015 Song et al. 2020a Ho et al. 2020; Nichol 2022 Dhariwal & Nichol 2021 undertake the reverse operation by initiating with Ty sampled from pure Gaussian noise and progressively denoising it reconstruct the image wp- Importantly, SDE has its corresponding "probability flow" Ordinary Differential Equation (PF ODE) (Song et al. a), which delineates a deterministic pathway that yields the same distribution p((x)) for Vt, thereby offering a more efficient sampling mechanism:
& Dhariwal 2021 Song al.) 2020b; Karras et al.
1 [u(x, t) (t)
dx.
dt,
(72)
where V log Pt (a) is referred to as the score function of (a)) (Hyvärinen 2005
Various ODE
solvers have been introduced to further expedite the sampling process utilizing Eq. 72 or minimiz-
ing the truncation error (Liu al. 2022 Jolicoeur- Martineeu et al. 2021 Karras et al. 2022 Lu
et al. 2022).
A pivotal insight within diffusion models is the realization that V log (x)) can be approximated by a neural network se (Iz, t), which can be trained using the following objective:
A(t) (x,, t) Vx, log pog
(73)
where A(t) represents the loss weighting and y denotes a training image.
B.2 CONSISTENCY MODELS
Drawing inspiration from DM theory, consistency models (CM) have been proposed enable single-step generation (Song et al., [2023] Song & Dhariwal 2024). Whereas DM incrementally denoises an image, c.g., via the PF ODE, CM denoted by to is designed map any point æ, at any given time along PF ODE trajectory directly to the trajectory's initial point æ; in single step.
CMs are usually trained through discretized time steps, so we consider segmenting the time span from [c. 7 into K 1 sub-intervals, with c being a small value approximating zero. Training of CM can follow one of two primary methodologies: consistency distillation (CD) or direct consistency training (CT). In the case of CD, the model So leverages knowledge distilled from a pre-trained DM 6. The distillation loss can be formulated as follows:
BR- -1j Ems NWy. in Mtadd fx+1), fo (#) tx)),
(74)
where fe- refers to the target model and 0 is computed through EMA of the historical weights of 0, 2t is estimated by the pre-trained diffusion model é through one-step denoising based on Et and d a metric implemented by either the &a distance or LPIPS (Zhang et al.) 2018).
Alternatively, in the CT case, the models So are developed independently, without relying on any pre-trained DM:
Ex-e B Eeeerraaat..eeeeeee... Mt) ((ta)
(x+12,t(+1); fo (y tx)),
(75)
where the target model fe- is set to be the same as the model fo in the latest improved version of the consistency training (Song & Dhariwal 2024), i.e. 0- = 0.
B.3 WEIGHT AVERAGING
The integration of running average of the weights by stochastic gradient descent (SGD) was ini- tially explored within the realm of convex optimization (Ruppert 1988 Polyak & Juditsky 1992). This concept was later applied to the training of neural networks (Szegedy et al. 2016). Izmailov et al. (Izmailov et al.] 2018) suggest that averaging multiple weights over the course of training can yield better generalization than SGD. Wortsman et al. (ooossstmnn et al. 2022 demonstrate the po- tential of averaging weights that have been fine-tuned with various hyperparameter configurations. The Exponential Moving Average (EMA) employs a specific form that uses the exponential rate ? as smoothing factor:
0, 70 (1 Om,
(76)
19

---

Published as a conference paper at ICLR 2025
where n I.... .,.............................. NN denotes the number of training iterations, @ represents the EMA model, and is initialized with 0g- 0o.
Practitioners often opt for advanced optimizers such as Adam (Kingma & Ba 2014) for different tasks and network architectures, which might reduce the need for employing running average. However, the use of E.MA has been noted to significantly enhance the quality of generation in early
DM studies (Song et al. 2020b Dhariwal & Nichol 2021; Nichol & Dhariwal 2021] Ho et al. 2020 Song et al. 2020a This empirical strategy has been adopted in most, if not all, subsequent research endeavors. Consequently, CM have also incorporated this technique, discovering EMA models that perform substantially better.
Recently, several works of more advanced weight averaging strategies have been proposed for Large Language Models (LLMs). Most of them focus on merging models fine-tuned for different down- stream tasks to create a new model with multiple capabilities (Ilharco al. 2022] Yadav al. 2024] Jin et al., 2022 Yu et al. 2023). Different from these approaches, our work aims to accelerate the model convergence and achieve better performance in standalone training process. Moreover, methodology we employ for determining averaging coefficients is novel, thereby distinguishing our method from these related works. Further details can be found in Sec. and Sec. 4
B.4 BASED METHODS FOR DIFFUSION MODELS
Search algorithms are widely used across various domains like Neural Architecture Search (NAS) (Real et al. 2019 Ning et al. 2020), where they are employed to identify specific targets. Recently, many works studies have set discrete optimization dimensions for DMs and utilized search methods to unearth optimal solutions. For example, Liu et al. (Liu et al. 2023a). Li et al. (Li et al.) 2023) and Yang al. (Yang et al., 2023) use search methods find the best model schedule for DMs. Liu et al. (Liu et al. 2223b) apply search methods to find appropriate strategies for diffusion solvers. However, these works do not involve modification to model weights during search and are only applicable to DMs.
C LCSC ALGORITHM
The detailed algorithm of LCSC is provided in Alg.
D ADDITIONAL EXAMPLES OF THE METRIC LANDSCAPE
We previously introduced the metric landscape of the DM and CD models in Sec. In this section, we extend our analysis by evaluating additional metric landscapes using the FID. Specifically, we incorporate an additional intermediate training iteration and explore the metric landscape the CT model. The comprehensive landscapes of the DM, CD, and CT models are depicted in Fig. 7
Since we use grid search get the performance and using interpolation get the smooth landscapes, we further provide some examples of the original performance heat-map in grids, shown in Fig.
E
EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
E.
EXPERIMENTAL DETAILS
E.
TRAINING SETTING
Since LCSC employs weight checkpoints, we endeavored to replicate the training processes of the baseline models.
For DM, we follow DDIM (Song et al. 2020a) for the evaluation on CIFAR10 (Krizeesky et al.) 2009 and iDDPM (Nichol & Dhariwal 2021) for the evaluation on ImageNet-64 (Deng et al,, 2009 To improve the inference efficiency, we adopt DPM-Solver (Lu et al. 2022). For CM (Song et al. 2023), we evaluate LCSC with both CD and CT on CIFAR-10 10 and ImageNet-66 64, and CT on implementation. We follow all the settings reported by the official paper except ImageNet-64, on which we decrease the batch size to 256 CM and 512 DM due to the limited resources. We apply LCSC at
LSUN datasets. For CIFAR-10 and ImageNet-64, We train models with
20

---

search process to Published as a conference paper at ICLR 2025
Algorithm Evolutionary Search for Combination Coefficients Optimization
Require:
θ (Onz: Ong: Ong. Ong ): the set of saved checkpoints untill training iter T
F: the metric evaluator.
Symbols:
P: The whole Population of model schedule.
CP: The Candidate Parents set of each loop, from which parent coefficients is selected.
NG: The Next Generation newly mutated from the parent coefficients in each loop.
0: A group of combination coefficients denoted as a {ai, ,
or θ: Equal to (1 -1 0,) + E Og0n,
Hyperparameters:
Epoch: Number of loops for the entire search process.
Mcp: Maximum size of the candidate parents set CP.
Iter: Maximum number of mutations in each loop.
Search Process:
1: 2
2: Initialize a group of coefficients Oiiiit with EMA weights
Pt PU {CCinst. F(amii θ))}
4: for = 1, 1,... ,Eoch do
5: NG t-
6: for 1, Iter do
7:
CP + {a, θ) ranks within the top min(MMr, P} Random Sample
8:
af, am
CP
9:
Onew f- Mutate Mutate(Crossooeer Crrssssversssss am))
10: NG + NG J {(Onew, Flanew 9))
11: end for
12: P+PUNG
13: end for
14: o² + arg min F(a θ) aEP 15: return a
different training stages. For LSUN datasets, we fine-tune the released official models and apply LCSC. We further train LCM-LoRA model following the official setting and use LCSC with the final checkpoints. Specifically, for any selected iteration, we utilize checkpoints from every predetermined interval of iterations within defined window size. Then find the optimal combination coefficients.
For DMs. were able reproduce the results reported in the original papers successfully. How- ever, for CMs, our training outcomes on CIFAR 10 were slightly inferior to those documented in the original papers. Additionally, due resource constraints, we opted for smaller batch size than the original configuration when training on ImageNet. In the results tables, we present both our training outcomes and the results reported the original papers, with the latter indicated as released. The specifics of the experimental setups are detailed below.
For the experiments with DM, utilize the DDIM (Song et al. 2020a) codebase https: 7/github. com/ erweng ddim) on CIFAR- 10, adhering to the default configuration set- tings. For 6mageNet-64, we employ the iDDPM (Nichol & Dhariwal 202T) codebase (https: ope.///eaaaaaaa wttp://oo/////c////c//caaaaaaa http:eannnnmmmoooedddddd fiussiiionnfoiooo setting the batch size to 512, the noise schedule to cosine, and maintaining other hyperparameters their default values. During sampling, the DPM-Solver (Lu et al. 2022) (ht /tttps://githui /hwpbht.thtutgg.g com/ LuChengTHU/ Htwppmmm------ sootmmssoverrrr is applied, conducting 15 timesteps of denoising using the default configuration for the respective setting. The reproduced results on par with slightly surpass those achieved using Euler inte- gration, as reported the original papers.
7/githab. ctm/
For the experiments involving CM, adhere closely the configurations detailed in Table of the original CM paper (Song et al. (023). On CIFAR-10, instead using the NCSNPP model (Song et al. 2020b) from CM's official implementation, opt for the EDM (Karras al.
21

---

Published as conference paper at ICLR 2025
Models Near 200K Training Iter
Models Near 800K Training Iter
1.-
8.9
0.7
0.6
0.3
0.6
20
0.0
0.1
15
0.2
0.3
0.5
0.6
08-
12
0.
12
11.
12
100
EMA Near 200K Training Iter
EMA Near 800K Training Iter
11
11.0
0.9
88
10.
8.5
10.0
0.3
82
0.0
95
411
4.3
488
0.
477
08
85
3.0-
Figure 6: Metric landscapes in grid for CD.
2022) architecture. This decision stems from the official CM code for CIFAR-10 being imple- mented in JAX while our experiments are conducted using PyTorch. For further details, refer to holttps://githhb.... com www.c///ppeeaicossssstttttoooooa
models cifarl0 This difference in implementation may partly explain the discrepancies between replication results and those ported Table of CM's original paper (Song et al. 2023). On ImageNet-64 and LSUN, we follow the official implementation, employing the ADM (Dhariwal & Nichol 2021 architecture. Due to resource constraints, we train the model with a smaller batch size of 256 instead of 2048 as used in the original study, resulting in worse outcomes. However, indicated in Tab. applying LCSC to models trained with a reduced batch size can still achieve performance comparable to models trained with larger batch size.
E.1 SEARCH SETTING
At each selected training iteration, historical weights are leveraged within designated window sizes and intervals. For CM, checkpoints have a window size of 40K with an interval of 100 on CIFAR- 10, and window size of 20K with an interval of 100 on ImageNet-64 and LSUN datasets. DM assigned a window size of 50K and interval of 200 for both datasets. FID calculation for DM on ImageNet-64 and CM on LSUN datasets utilizes a sample 5K images, while 10K images are used for all other configurations. An evolutionary search spanning 2K iterations is applied consistently across all experimental setups.
E. TEXT-TO-IMAGE TASK
For text-to-imann toaeiiageeeamoooo task, we fine-tune a LoRA based on the Stable Diffusion v1-5 model (Rombach al. 2022) CC12M dataset. We use batch size of 12 and train the LoRA for 6k steps. For LCSC, we use the checkpoints saved between 4k and 6k steps with an interval of 20. When sampling, insert LoRA Dreamshaper-7, which is fine-tuned version of Stable Diffusion v1-5, and use Ik
22

---

Published as a conference paper at ICLR 2025
Model Trairing
1.0
1.0
&
16.
0.5
0,5
0.0
o
50
11.0
-0.5
0.0
0.5
1.0
-1.0
-05
0.0
0.5
1.0
110
00.5
0.0
0.5
1,0
Br.
6g.
Br.
EMA 50K Training
200K Training
Near BOOK Trinning
A,
380
-0.5
-1.0
-1.0
-1.0
-1.0
1.0
1.0
-0p
Metric landscapes for DM.
Models Near 50K Training Iter
Models Near 200K Training Iter
Models Near BOOK Training Iter
1.0
1.0
1.0
0.5
0.5
0.0
-0.5
-0.5
12
0..5
324
-1.0
-1.0
33
-1.
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
C.0
0.5
1.0
-1.0
-0.5
0.0
6.5
1.0
-
0g
0, 0,
Training
Training
1.0
1.0
as
0.5
0.5
n.c
85
-0.5
-0.5
-1.0
-0.5
0.0
0.5
1.0
-10
-0.5
0.0
1.0
-.00
-05
0.0
0.5
100
fg
9,
(b) Metric landscapes for CD.
EMA Near 50K Training Iter
Model Near 200K Training Iter
Model Near 800K Tiairing Iter
1.0
1.0
1.0
0.5
0.5
0.5
212
LL
0.0
0.0
0.5
213
0.5
333
0.5
-1.0
265
-1.0
-10
33.
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
D.G
0.5
1.0
-1.0
-0.5
0.9
6.5
1.0
0.
-
6. f.,
8, -,,
Training
1.0
4,
1.0
10.
400
0.5
0.5-
0.5
9,
100
0.0
9,
:3
334
-1.0
-0.5
0.0
1.
1.0
0.5
0.0
0.5
1.0
1.0
0.5
0.0
0.5
1.0
6x.
By.
(c) Metric landscapes for CT.
Figure 7: Metric landscapes for DM, CD and CT CIFAR-10.
samples from CC12M to calculate FID. Finally, we randomly sample 10k samples from MS-COCO dataset evaluate the search results.
23

---

Published as a conference paper at ICLR 2025
Table 8: Search time consumption per search iteration (in seconds). Results marked with * denote the CPU time cost, which is excludable from the overall time cost through parallel processing. "U- Net" refers to the denoising sampling process, "Inception" refers the computation of inception features, "Merging" refers to the averaging of weights, and "FID" refers to the computation of FID statistics.
Model
Dataset
U-Net
Inception
Merging*
FID*
CIFAR-I0
7.34
4.28
5.57
2.49
CM
ImageNet-64
34.7
4.32
12.0
2.37
CIFAR-I0
40.9
4.39
4.61
2.49
DM
ImageNet-64
114.5
2.24
13.8
2.53
For baselines, we find that the performance is sensitive the scale of LoRA. Therefore, we first conduct coarse scan to determine the approximate range of the optimal LoRA scale, followed by a fine sweep. We find 0.15 the optimal scale for LoRA and use it as the baseline.
E.2 TWO-PHASE CONVERGENCE
Fig. shows that the generation quality convergence of DM and CM can be divided into two phases. The initial phase is relatively brief, during which DM and CM rapidly acquire the capability to generate visually satisfactory images. In contrast, the second phase is characterized by a slower optimization of models, focusing on the enhancement of sample quality. It is noteworthy that majority of training iterations belongs to the second phase.
FID-iter curve CD CIFAR-10
FID-iter curve of DM CIFAR-10
45
329
30
150
Figure 8: Training curves CIFAR-10 with CM and DM. PI, P2 represent the first phase and the second phase.
10.
109
4.5-
18
4.2
7
1.
P1
P2
3.6
3,8
40 50 60
10
20
70
Training Iteration
Training Iteration
E.3 SEARCH COST ESTIMATION
We detail the estimation of search costs, covering both CPU and GPU time consumption. Saved checkpoints are loaded into CPU memory and then transferred to the GPU after averaging. Conse- quently, CPU time comprises the duration for merging weight checkpoints and calculating the FID statistic. Mcanwhile, GPU time is dedicated to the sampling and evaluation processes involving the diffusion U-Net and the Inception network. All experiments are performed on single NVIDIA A100 GPU, paired with an Intel Xeon Platinum 8385P CPU.
First, we profile the durations of each function across all experimental settings and document the time consumed in single search iteration in Tab, 8 We note that the CPU time cost is significantly lower than that on the GPU. Therefore, while conducting sampling with the current merged weights on the GPU, we can simultaneously perform the FID statistic computation from the previous itera- tion and merge the checkpoints for the next iteration on the CPU. result, the CPU time cost can be effectively excluded from the overall time cost.
Next, we examine the overall training and search time costs as outlined in Tab, 9 and we delve into the speedup ratio detailed in Tabs. 10 and []] Given that DMs necessitate multiple timesteps for sampling. the overall search cost for DM proves to non non- nog-negligibbl when compared to the training cost. This results in no observable speedup in convergence upon applying LCSC. However, this search cost is considered manageable and warranted for the anticipated improvements to the final model convergence.
24

---

Published as a conference paper at ICLR 2025
Table 9: Overall search time and training time consumption (in hours).
Search
Training
Model
Dataset
Iteration
Time
Iteration
Batch Size
Time
CIFAR-10
2K
6.45
800K
512
818
CD
ImageNet-64
2K
21.7
600K
2048
7253
CIFAR-10
2K
6.45
800K
512
640
CT
ImageNet-64
2K
21.7
800K
2048
7040
10FAR-10
2K
25.2
800K
128
29.3
DM
ImageNet-64
2K
64.8
500K
512
372
Table 10: Accurate training speedup of consistency models CIFAR-10. The speedup is compared against the standard training with 800K iterations and 512 batch size.
Model
Method
Training Iter
Batch Size
Speedup(t)
EMA
800K
512
50K
512
14.27 x
L.SCC
250K
512
3.12x
100K
128
25.55
(released)
800K
512
CT
400K
512
1.96x
LCSC
450K
128
6.64x
Table 11: Accurate training speedup of consistency models on ImageNet-64. For CD, the speedup is compared against standard training with 600K iterations and 2048 batch size. For CT, the speedup is compared against standard training with 800K iterations and 2048 batch size.
Model
Method
Training Iter
Batch Size
Speedup(t)
EMA (released)
600K
2048
150K
256
29.20 x
LCSC
300K
256
15.27 x
620K
256
7.55x
EMA (relcased)
800K
2048
CT
600K
256
10.33
LCSC
1000K
256
6.27x
In contrast, for CMs, the search cost is substantially lower than the training cost, which often results markedly faster convergence when combining training with LCSC than with training alone. The precise speedup ratios for CM training are detailed in Tabs. 10 and 11 It should be noted that the figures in these tables may exhibit slight discrepancies from those reported in Sec. 5]due to the latter being profiled in an environment with more variables and potential fluctuations.
E.4 HYPERPARAMETER STUDY
The hyperparameters of LCSC includes the window size (nı - n1) for retrieving the historical weights, the interval (ng 74-1) for two adjacent weight checkpoints, the number of samples for computing FID and the number of search iterations, and the number of timesteps for DM.
Tab. 12 illustrates the impact of varying the number of samples, search iterations, and the size of the interval. We observe that increasing the number of samples and search iterations leads to higher performance, though this comes at the expense of increased search cost. In practice, find that 10K
25

---

Published as a conference paper at ICLR 2025
Table 12: Hyperparameter study of LCSC with (a) DM and (b) CM. The models are trained on CIFARIO with 250K iterations. We evaluate three values of each hyperparameter and compare them with our adopted setting highlighted gray. The varied hyperparameter in bold. For DM the window size is 50K, for CM it is 40K.
(a) Results DM
(b) Results CM
Samples
Search Iters.
Interval
FID(C)
IS(t)
Samples
Search Iters.
Interval
FID(.)
IS(t)
10K
2K
200
3.87
9.27
10K
2K
100
2.76
9.71
2K
2K
200
9.23
9.11
2K
2K
100
3.40
9.39
5K
2K
200
4.04
9.09
5K
2K
100
2.79
9.66
10K
1K
200
4.01
9.17
10K
1K
100
3.03
9.57
10K
4K
200
3.79
9.28
10K
4K
100
2.69
9.70
10K
100
3.82
9.15
10K
200
2.84
9.71
10K
4.41
9.11
10K
2.71
9.79
Table 13: Hyperparameter analysis of LCSC, supplementing the study in Tab. 12 Models trained on CIFARIO for 250K iterations. assess three different values for each hyperparameter, contrasting these with chosen setting, which is highlighted in gray. The hyperparameter under variation is indicated in bold. For DM, the interval between checkpoints set to 200, whereas for CM, it is 100. We perform 2K search iterations and sample 10K images compute the FID score at each iteration.
Method
Window Size
NFE
FID(L)
IS(1)
40K
2.76
9.71
CM
10K
2.89
9.60
50K
2.73
9.66
50K
15
3.87
9.27
10K
15
4.41
9.11
DM
30K
15
3.82
9.15
50K
7
3.91
9.10
50K
3.84
9.17
samples and 2K search iterations can efficiently identify strong models. With fixed window size, the interval between two checkpoints determines the dimension of the search space. Our findings suggest that search performance generally improves as the search space expands, but limiting the search dimension to fewer than 200 can detrimentally affect search performance.
For the impact of window size (nK 71) and sampling NFE of DM, the findings are detailed in Tab. 13 The results reveal that, despite earlier models being further from convergence, sufficiently large window for accessing historical weights proves advantageous. Furthermore, conducting a lower NFE during the search in the DM context results in similar reduction in FID but smaller improvement in IS. This suggests that the search output's generality across different metrics di- minishes when the generated samples during the search are less accurate, i.e., exhibit significant truncation error.
E.5 DETAILED RESULTS OF EMA RATE GRID SEARCH
For the final training model of each experimental configuration, conduct comprehensive sweep across a broad range of EMA rates, presenting the optimal outcomes in Sec.5) The exhaustive results are detailed in Tab. 14 In the majority of scenarios, the default EMA rate employed in the official implementations of our baseline models (Song et al.] 2023 Ho et al. 2020 Nichol & Dhariwal, 2021) yields slightly inferior performance compared to the best EMA rate identified. Nevertheless, the findings demonstrate that LCSC consistently surpasses all explored EMA rates, underscoring the limitations of EMA strategy for weight averaging.
26

---

Published as a conference paper at ICLR 2025
Table 14: EMA rate grid search outcomes. Asterisks (*) indicate the results using the default rates from the official paper. Listed FID scores for each EMA rate correspond to fully trained models: CIFAR-I 10 models at 800K iterations; ImageNet-64 CD/CT/DM models 600K/1000K/500K iter- ations, respectively. Iteration counts for models employed by LCSC are shown in parentheses.
EMA rate
Model
Dataset
LCSC
0.999
0.9995
0.9999
0.999943
0.99995
0.99997
0.99999
CIFAR-10 10
4.35
4.13
3.66*
3.56
3.58
3.51
3.54
2.42 (800K)
ImagoNet-64
7.45
7.38
7.19
7.17*
7.17
7.22
7.41
5,54 (620K)
CIFAR-10
9.80
9.78
9.70*
9.70
9.69
9.71
9.77
8,60 (400K)
ImageNet-6d
15.7
15.7
15.6
15.6*
15.6
15.6
15.7
12.1 (600K)
5.73
5.18
4.16*
3,99
3.96
4.04
5.04
3.18 (800K)
DM
ImageNet-64
23.1
20.3
19.8
19.0
18.1
18.1
18.5
15.3 (500K)
Table 15: The comparison of different formulations for base checkpoints on CIFAR-10 dataset. LCSC-Diff stands for the formulation we use in our main experiments. LCSC-Direct stands for using the checkpoints themselves the weighted base.
Model Method
Training Iter
Batch Size
NFF
FID(L)
IS(t)
Prec.())
Rec.(f)
Speed(1)
50K
512
3.18
9.60
0.67
0.58
~14
250K
512
2.76
9.71
0.67
0.59
~3.1x
LCSC-Direct
800K
512
2.42
9.76
0.67
0.60
800+40K
512
2.38
9.70
0.67
0.60
100K
128
3.34
9.51
0.67
0.57
~3x
CD
50K
512
3.10
9.50
0.58
14x
250K
512
2.66
9.64
0.67
0.59
<31x
LCSC-DII
800K
512
2.44
9.82
0.67
0.60
800+40KK
512
2.50
9.70
0.68
0.59
100K
128
3.21
9.48
0.66
0.58
23x
400K
512
8.60
8.89
0.67
0.47
1,9
LCSC-Direc
800+40K
512
8.05
8.98
0.70
0.45
450K
128
8.33
8.67
0.69
0.44
~
CT
400K
512
8.89
8.79
0.67
0.47
~99
LCSC-Diff
800+40K
512
7.05
0.70
0.45
450K
128
8.54
8.66
0.69
0.44
~x
E.6 DIFFERENT FORMULATION OF THE BASE CHECKPOINTS
As discussed Sec. we subtract the first checkpoint from each checkpoint and use these dif- ferences as the weighted base. In this section, we explore an alternative approach for defining the base checkpoints: using all the checkpoints themselves as the base. this case, the final combined weight is:
K
a Og
(77)
We apply this formulation in search, with the results presented in Tab. [nd see that the difference search method outperforms the checkpoint search method all cases on the ImageNet-64. On CIFAR-10, the performance of the methods similar with each other.
and Tab. [6
We can
E.7 EVALUATION WITH OTHER METRICS
E.7.1 EVALUATION WITH FCD AND KID
In this section, we present the results of FCD and KID using CT method on CIFAR-10 and ImageNet-64 dataset. The results are shown in Tab. 17
27

---

Published as a conference paper at ICLR 2025
Table 16: The comparison of different formulations for base checkpoints on ImageNet-64 dataset. LCSC-Diff stands for the formulation we use in our main experiments. LCSC-Direct stands for using the checkpoints themselves the weighted base.
Model
Method
Training Iter
Batch Size
NFE
FID(C)
IS(1)
Prec.(t)
Rec.(f)
Speed(")
300K
256
5.71
41.8
0.68
0.62
~15x
LCSC-Dired
600+20K
256
5.54
40.9
0.68
0.62
~7.6
CD
300K
256
5.51
39.8
0.68
~15x
LCSC-Diff
600+20K
256
5.07
42.5
0.69
0.62
~7.6
600K
256
12.1
35.1
0.67
0.54
10.4x
LCSC-Direct
800K
256
11.1
35.7
0.65
0.57
~6.3
CT
600K
236
10.5
36.8
0.66
0.56
~ 10.4x
LUSC-DRI
800K
256
9.02
38.8
0.68
0.55
7.33
Table 17: Evaluation results with FCD and KID metries on CIFAR-10 and ImageNet-64 datasets.
(b) on ImageNet-64.
Method
Training lter
Ratch Size
FID(U
PCD())
KID())
Method
Training Iter
Batch Size
FID()
PCDX)
KID())
CT
400k
512
12.1
43.0
7.47e-3
CT
600k
256
16.6
49.8
9.23c-3
5.14e-3
CT
80ik
256
15.8
47.5
8.69e-3
LCSC
450k
128
8.54
27.2
4.65e-3
CT
800k
2048
13.1
47.5
8.55c-3
LCSC
400k
512
8.89
34.4
3,96e-
LCSC
600k
256
10.5
38.8
4.4le-3
LCSC
800+40K
512
7.05
24.9
2.90e-3
LCSC
800k
256
9.02
3.87e-3
E.7.2 EVALUATION WITH FID ON TEST DATASET
In this section, we report the FID calculated on the test dataset to validate that LCSC does not over- fit the training data. Results are shown in Tab. [8] We that LCSC also achieves significant improvement on test FID, indicating its generalization ability across different data.
E.8 SEARCH RESULTS WITH STABLE DIFFUSION MODELS
To further demonstrate the effectiveness of LCSC in enhancing DMs, we conduct experiments on Stable Diffusion (Rombach al.) (022) checkpoints. Specifically, fine-tune the Stable Diffusion v1-5 model on CC12M using LoRA for 20k iterations. We then apply LCSC to the saved check- points intervals of 100 iterations. The results, presented in Tab. [9. show that LCSC achieves a significant improvement compared to the released Stable Diffusion checkpoints with the same sampling NFE.
To investigate whether LCSC can accelerate the Stable Diffusion model, we further test PickScore (Kirstain et al. 2023) between the LCSC model with 10 NFE and the Stable Diffusion model with 15 NFE. The results are 0.49 and 0.51 for Stable Diffusion and LCSC. respectively, with 57% winning rate for LCSC. This further demonstrate the ability of LCSC accelerate inference speed for diffusion models.
F MORE INSIGHTS AND ANALYSIS
F.1 ANALYSIS OF SEARCH PATTERNS
F.1.1 MORE EXAMPLES OF SEARCH PATTERNS
We demonstrate search results ImageNet-64 in Fig. which share similar patterns with results on CIFAR-IO (refer to Fig. 5). Additionally, Fig. shows that the checkpoints assigned with larger coefficients often has lower FID than the checkpoints with small coefficients.
1.2 SEARCH PATTERN OF DIFFERENT RANDOM SEED
We compare search patterns across different random seeds. As shown in Fig. although each seed's dominant coefficients map to distinct subsets weight checkpoints, they produce similar outcomes in terms of FID) provided in the caption. These results suggest multiple high-quality
28

---

Published as a conference paper at ICLR 2025
Table 18:
Evaluation results on CIFAR-10
Method
Training Iter
Batch Size
FID(L)
test FID(L)
CD
800k
512
3.66
5.88
CD
840k
512
3,65
5,85
LCSC
100k
128
3.21
5,48
LCSC
250k
512
2.66
4.88
LCSC
800k
512
2.44
4.70
LCSC
800+-00k
512
2.50
4.75
Table 19: Results of LCSC on Stable Diffusion Checkpoints.
NFE
FID
PKS
WR@PKS
CLIIP Score
LCSC
16.30
0v53 SDW0053 LoRA)
55% SDV56 (v.s. LoRA) 44%
26.69
SDv1-5
17.55
0.47
LoRA tuning
17.05
0.47
4.5%
26.61
LCSC
10
16.68
(v.s. SD)/0.51(yss. LoRA) 64% (v.ss.s. 3DV53%((v. LoRA)
26.61
SDv1-5
10
18.16
0.41
36%
26.57
LoRA tuning
10
17.35
0.49
47%
26.56
Results ImageNet-64 with
Results ImageNet-64 with
Results on ImageNet-64 with CT
Weights
2.0
Weights
0.099943
0.999943
0.999943
1.0
0.1
0.5
0.0
-1.0
-6.2
15-
-2.0
45
46
27
48
49
50
60.0
65.5
61.0
61.5
62.0
38.0
56.5
56.0
59.5
65.0
Training (eratton(11Kk)
Training (taratenixx0kk
Training (trattinnxx00))
Figure 9: Visualization of weight combination coefficients obtained using LCSC compared those from the default EMA on ImageNet-64.
Results on CIFAR-10 with CD
Results on ImageNet with CT
9.0
8.5
8.0
22.2
7.0
6.5
22.0
6.0
5.5-
21.8
5.0
21.6
-0.2
0.0
0.3
-2
Coefficient
Coefficient
Figure 10: FID of checkpoints with varying magnitude of coefficients.
basins exist within the weight checkpoint subspace, and LCSC converges to one of them depending on the randomness. Previous study demonstrates that local minimal in networks are connected by simple curves over which losses are nearly constant (Garipov et al. 2018). To investigate whether solutions found by LCSC are also connected we average the search patterns obtained from multi- ple random seeds and evaluate the performance of the resulting model. As shown in Fig, 12 the averaged search pattern becomes homogeneous and contains smaller coefficients, yet it achieves comparable performance, as detailed in the figure caption. This observation suggests that the search results of L.SC may also be connected by low-loss curve.
29

---

Published as a conference paper at ICLR 2025
Searched Weights with Different Random Seeds
seed
0.3
seed
0.2
0.1
0.0
0.11
0.2
0.3
21
24 Training Iteration (xattion(xxxn X10K
22
23
25
Figure 11: Visualization of linear combination coefficients obtained by LCSC with different random seeds using CD on CIFARIO. Their FID scores are: 2.76 (seed 1) and 2.69 (seed 2).
Averaging
Averra between
01
0.0
0.0
0.0
50 200 450
50
55
Trairing Ieeaaoonn22 060
Training Beradoon<200)
Training Reradoon>210))
Figure 12: Visualization of averaged coefficients obtained by LCSC across different random seeds using CD on CIFARIO. Their FID scores are: 2.76 (left), 2.70 (middle), 2.83 (right).
F.1.3 THE CRITICAL ROLE OF SEARCH SPACE CONFIGURATION
As explored in Sec. restricting averaged weights to the convex hull of candidate checkpoints, where all combination coefficients are non-negative and their sum equals 1, might limit search effi cacy. To test this hypothesis, we perform searches under convex combination conditions by clipping coefficients to non-negative values and normalizing their sum to Tab. 20 presents these findings, with "w/o" indicating restriction coefficients (allowing values below zero) and "w" represent- ing the convex hull restriction. The results clearly show superior performance without the convex restriction, underscoring its limiting effect search outcomes. Furthermore, we discover that nor- malizing the coefficient sum to is vital for effective search exploration, a practice we continue even after lifting the convex condition, detailed in Eq. (
F.1.4 FORMATION OF THE SEARCH PATTERN
We observe that the search pattern is established during the early stage of the search, while the later stage primarily amplifies it to certain magnitude, shown in Fig. 13
30

---

Published as a conference paper at ICLR 2025
Table 20: Performance of LCSC with or without restriction the convex hull of all saved check- points. "w/o" in the column "Restriction" is our default setting in main experiments, while "w" means the restriction holds.
Model
Method
Training Iter
Batch Size
NFE
Restriction
FID(C)
IS(t)
w/o
2.76
9.71
LCSC
250K
512
W
3.38
9.36
w/o
3.56
9.35
DM
LCSC
350K
128
W
3.59
9.29
Searth Results 2000 ier Ofina
Search Results 1000
Reeutss
Search Reouts at 100 ber
2-
100
125
350
175
200
Figure 13:
Visualization of coefficient pattern at different number of search iterations. The experi- ment is conducted on ImageNet using CT.
F.2 REGULARIZED EVOLUTIONARY SEARCH
conduct additional experiments to evaluate the impact of regularization. Specifically, during the search process, clip all coefficients to be below 1, thereby constraining the search to restricted space. Using the same initialization and random seed, Fig. [4] shows that LCSC produces smaller and more homogeneous coefficients with regularization compared to those obtained without it. However, as noted in the figure caption, applying regularization results worse performance, suggesting that constraining L.CSC smaller and more homogeneous coefficients is not advantageous.
F.3 CONVERGENCE CURVE OF LCSC
Fig. 15a illustrates the convergence curve of the models searched by LCSC. At each point, corre- sponding to a specific number training iterations, we perform LCSC with 2K search iterations using the most recent checkpoints available at that point. The results indicate that the number of training iterations increases, the models searched by LCSC converge to progressively lower FID values. Notably, the convergence curve of LCSC exhibits similar trend to that of the EMA model's
31

---

Published as a conference paper at ICLR 2025
Searched Coefficients with Regularization
Searched Coefficients without Regularization
2-
M aa
-1
-3
58.5
59.0
58.0
59.5
Training (x)1ation(x11))
Training Iteration(x11k)
(a) Visualization of searched coefficients with regu- larization.
(b) Visualization of searched coefficients without reg- ularization.
Figure 14: Visualization of searched coefficients with and without regularization. Respective FID/IS/Prec/Rec are: 13.40/33.4/0.64/000 (left), 10.3/36. 0113366800666/0.566 (right).
Training Curve with LCSC CIFAR-10 with CD
Snarching Curve LCSC CIFAR-10 with CD
5.0
3.8
4.5
4.0
3.4
3.5
3.2
3.0
3.0
2.8
2.5
10
20
30
60
0
500
1000
1500
2000
Trainino (x0)))
Search Iteration
(a) Training convergence curve with LCSC,
(b) Scarch convergence curve of LCSC.
Figure 15: Convergence curves of LCSC on CIFAR-10 with CD.
convergence curve. However, LCSC consistently achieves lower FID values compared to the EMA model. highlighting its enhanced ability to effectively merge historical checkpoints. Additionally, Fig. 15b shows the convergence curve of LCSC iteself. This curve consistently reduces as the num- ber of search iteration increases.
F.4 DISCUSSION ON THE TRAINING VARIANCE OF DM AND CM
In Sec. mentioned that the training objectives of DM and CM tend to introduce substantial variance in gradient estimations. We further discuss the potential reasons or hypothesis of such phenomenon as follows.
F.4.1 UNBIASED ESTIMATION AS OBJECTIVE
The objectives used in DM and CT not accurate for any single batch. Instead, they an unbiased estimation. Specifically, for DM, the neural network learns the score function at any (æ,, f). denoted as sæ(x,, t). The ground truth score function V (og,)( ) is given (Song al. 2020b):
Vog (x:)) =E, -
lo.y + OrE xx)) 02
(78)
This formula implies that every sample y in the dataset contributes to the ground truth score function at any (I,,t), However, the training objective Eq. (2) provides only an unbiased estimation of the ground truth score Eq. (78). At every iteration, the current target for the network is not identical to the ground truth score function but is determined by the sample y randomly drawn from the dataset and the noise level oe. Thus, even if sp(x,, t) exactly matches Vp,(x,) for v( t), its gradient estimation using a mini- batch data does not converge to zero. The approximation of ground truth score function only be obtained through the expectation over many training iterations.
32

---

Published as a conference paper at ICLR 2025
For CT, the situation is very similar. To simulate the current ODE solution step without teacher diffusion model, Song et al. (Song et al. 2023) utilize the Monte Carol estimation -= to replace under the noise schedule from EDM (Karras al. (022) as denoted in Eq. (4) Therefore, the objective is also not accurate for any single batch of data and the model has to learn to fit the target model output after solving the current step with the expectation of score function among many training iterations. These types of objectives, which are not accurate for any single batch, introduce high variance to the gradient estimation.
the ground truth score function V log p. (a.)) = Eyy-pan. -Ny, 1- |x:)
F.4.2 MODEL ACROSS DIFFERENT TIME STEPS
For both DM and CM, the neural network has to be trained at various time steps. Since the model input and output follow different distributions, the gradients at different time steps may conflict with each other. This phenomenon of negative transfer between timesteps has been studied in several previous work, where they train different diffusion neural networks at different timesteps and achieve better performance (Hang et al. 2023] Go et al. 2024). The misalignment between the objectives of different time steps may also contribute to the high variance in the gradient estimation.
F.4.3 ERROR ACCUMULATION OF CM
For CM, the model learns to approximate the output of the target model at the previous step. This could potentially introduce the problem of error accumulation (Berthelot et al. 2023). Consequently, any noise introduced during training early time steps is likely to lead to inaccuracies in the target model, which may be magnified in subsequent timesteps. This property of CM amplifies the high training variance, particularly for one-step sampling.
G FUTURE WORK
LCSC represents novel optimization paradigm, indicating its potential for widespread application. We recommend future investigations focus on three key areas:
Expanded Search Space: Presently, LCSC applies uniform coefficient across an entire model. Considering that different model layers might benefit from distinct combination coefficients, partitioning model weights into segments for unique coefficient assignments could enhance the search space. For DMs, adopting variable coefficients across different timesteps may also offer further improvements.
Efficient Optimization Methods: The current reliance on evolutionary methods, characterized by their dependency on randomness, limits efficiency and risks convergence to local optima. Investigating more effective optimization strategies presents a promising avenue for enhancing LCSC.
Broader Application Scope: While the initial motivation for LCSC stems from managing the high training variance observed in DMs and CMs, its utility is not confined to these models. Exploring LCSC's applicability other domains, such fine-tuning language models addi- tional vision models, could unlock new performance gains.
H VISUALIZATION
In this section, visualizations of images generated by LCM-LORA (Fig. 16. DM (Figs. 17 and 18) CD (Figs. a9 20). and CT (Figs. 2T and 22) are presented. Many images produced by the EMA model are observed to be similar to those produced by the model derived from our LCSC when using the same noise input. This is expected given that both models are based on weights from the same training cycle. To more clearly highlight the distinctions between EMA and LCSC, we generate 50K images from EMA and LCSC using the same set of noise inputs, order the image pairs according their Euclidean distances in the inception feature space, and randomly select the images with large distances. In general, images generated by the LCSC model are found to display enhanced sharpness, diminished noise, and more distinct object representation well as details.
33

---

Published as a conference paper at ICLR 2025
(a) Vanilla training
(b) LCSC
Vanilla training
(d) LCSC
(e) Vanilla training
LCSC
(g) Vanilla training
LCSC
Figure 16: Examples the images generated by LCSC and vamilla training. The prompts for the images are 'A large passenger airplane flying through the air', "A brown purse is sitting green bench', "A man getting a drink from a water fountain that is toilet', 'A picture of man playing violin in kitchen' respectively.
(a) EMA model (FID: 4.16).
(b) Model obtained through LCSC (FID: 3.18).
Figure 17: Generated images from DM trained on CIFARIO over 800K iterations.
34

---

Published as a conference paper at ICLR 2025
(a) EMA model (FID: 19.8).
(b) Model obtained through LCSC (FID: 15.3).
Figure 18: Generated images from DM trained on ImageNet-64 over 500K iterations.
(a) EMA model (FID: 3.66).
(b) Model obtained through LCSC (FID: 2.42).
Figure 19: Generated images from a CD model trained on CIFARIO for 800K iterations.
35

---

Published as a conference paper at ICLR 2025
(a) EMA model (FID: 7.30).
(b) Model obtained through LCSC (FID: 5.54).
Figure 20: Images from CD model trained on ImageNet-64 for 600K iterations.
(a) FMA model (FID: 12.08).
(b) Model obtained through LCSC (FID: 8.60).
Figure 21: Generated images from CT model trained on CIFAR 10 for 400K iterations.
36

---

Published as a conference paper at ICLR 2025
(a) EMA model (FID: 16.6).
(b) Model obtained through LCSC (FID: 12.1).
Figure 22: Images from CT model trained on ImageNet-64 for 600K iterations.
37


Reviews:
[{"review_id": "aAWxDJE84u", "reviewer": "Reviewer_K43d", "summary": "This paper proposes LCSC (Linear Combination of Saved Checkpoints), a method to improve the performance and efficiency of diffusion and consistency models. The underlying idea is to use an optimal linear combination of model checkpoints saved during training. These optimal coefficients are determined using an evolutionary search method. The authors demonstrate that this approach can not only decrease the training cost but also enhance the performance of pre-trained models. The experiments which were performed on CIFAR-10 and ImageNet-64 provably demonstrate that this approach results in an increase in training time and improvements in evaluation metrics.", "questions": "- Currently, the experiments seems to be limited to image generation tasks. Would it be possible to demonstrate this to another diffusion task such as audio or video?\n- Have you studied how these coefficients change over the course of training? There perhaps maybe an interpretability study to understand the importance of different stages of training on the best performance\n- On a similar front, can it be extended to layers? That is, composition of weights on layers may differ from the composition of overall weights and perhaps lead to a better result?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The simple methodology presented seems to have a well-motivated theoretical basis, and is quite effective in improving the training of diffusion and consistency models. Further, this method does not require backprop and side steps the need for differentiable loss functions through the evolutionary search.\n- The analysis presented on the method along with the visualizations of the landscape to demonstrate that optimal model weights lie in basins which are inaccessible by optimization but may be accessible through checkpoint averaging.\n- It has clear value in reducing compute time and improving model performance as demonstrated through the experiments. Seems like there's speedups by an order of magnitude which is quite impressive.", "weaknesses": "See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iNWBm5HBga", "reviewer": "Reviewer_odMx", "summary": "The paper proposes a method named LCSC for training diffusion models by linearly combining the intermediate weight checkpoints using an evolutionary algorithm. LSCS can significantly reduce the training cost of a diffusion model without sacrificing the generation quality. In addition, a pre-trained diffusion model can achieve better performance by fine-tuning with LCSC with few training iterations. The numerous experiment results are provided to demonstrate the effectiveness of LCSC in different datasets.", "questions": "(1) Can LCSC reduce the training cost of DM (not CM)? I only find experiments about reducing training cost with consistency distillation and consistency training but not vanilla DDPM. \n\n(2) It seems that most experiments are based on CM. Do you think LCSC is more suitable for CM and CD compared with DM? And why?\n\n(3) Could you provide the CLIP-score for text-to-image generation?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) The motivation is clear and easy to understand. The authors also provide theoretical analysis about why LCSC is better than EMA.\n\n(2) The method is simple and straightforward, which uses the evolutionary search to linearly combine checkpoint weights. It seems that LCSC can be easy to implement and applied with different diffusion models.\n\n(3) As shown in the experiment results, LCSC can significantly reduce the training cost of a diffusion model and enhance the pre-trained diffusion models. \n\n(4) The authors provide a lot of experiment results to study LCSC from different aspects.", "weaknesses": "(1) It seems that the experiments about reducing the training cost only focus on datasets with small resolution (CIFAR-10 and ImageNet-64). I understand that the cost for training a diffusion model on a dataset with large resolution (e.g, LSUN-bedroom, COCO) from scratch is more expensive. I believe that such a set of experiments can make the LCSC more attractive.\n\n(2) In my opinion, the experiment results about text-to-image generation are not convincible enough to validate the effectiveness of LCSC. The number of training iterations for fine-tuning LCM with LoRA is very small, which is only 6K with the batch size of 12. The authors should use a larger number of training iterations, or show that the generation quality of LCM will not improve with further training. I'm wondering whether the experimental settings for evaluating LCM with LCSC is suitable, since the generated images (Figure 2 and Figure 10) are in anime style while CC12M and MS-COCO are not. Besides the LCM, the authors could also fine-tune the pre-trained Stable Diffusion v1.5 for a few iterations with LCSC to see if LCSC can further enhance it.\n\nIn conclusion, I think LCSC is a good work for its motivation and method. While the experiments could be further improved and organized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c1eLvHgIRQ", "reviewer": "Reviewer_Enc2", "summary": "The authors introduce a novel approach to enhance the performance and reduce the training costs of Diffusion Models (DM) and Consistency Models (CM) by learning a linear combination of saved checkpoints using evolutionary algorithms. The paper builds upon existing theoretical frameworks, demonstrating that Exponential Moving Average (EMA) models converge faster than their last-tier counterparts under specific conditions. Through a motivational experiment involving a grid search over numerous linear combinations of three checkpoints (with and without EMA), the authors reveal the existence of superior solutions compared to the traditional EMA approach. The proposed method, termed Linear Combination of Saved Checkpoints (LCSC), is further validated experimentally, showing significant reductions in training costs via fewer iterations and smaller batch sizes, as well as improvements in generation quality when applied to fine-tuned checkpoints.", "questions": "1. Potential Overfitting of Weights (Fig. 8): The weights depicted in Fig. 8 exhibit large absolute values (greater than 6), which might indicate overfitting. Could the authors address the following points?\n- Regularization: Were any regularization techniques employed to constrain the magnitude of the coefficients?\n- Interpretability: What is the theoretical or intuitive justification for assigning such large weights to certain checkpoints?\n2. Convergence Curves Post-LCSC Application: Visualizing the convergence behavior after applying LCSC can provide a clearer understanding of its impact on training dynamics. Including convergence curves would help illustrate how LCSC influences the training process compared to baseline methods like EMA.\n3. Convergence Curve of LCSC itself. How many iteration of evolution does it undertake before achieving a satisfactory solution?\n4. Add experiments for further assessing the robustness of LCSC-Derived Coefficients.\n5. Enhancing Interpretability of Coefficients: Understanding the role and significance of each coefficient can aid in demystifying the model's behavior. Implementing constraints or regularization techniques to limit the magnitude of coefficients may improve their interpretability. Alternatively, providing a theoretical justification or empirical analysis explaining the distribution of coefficient values can enhance comprehension.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Impressive Experimental Results: The experiments, particularly those presented in Tables 1 and 2, convincingly demonstrate the efficacy of LCSC for both CD and CT, showcasing significant reduction in training costs.\n2. Innovative Findings: Figures 4 and 6 highlight the potential for significantly better linear combinations of checkpoints than the naive EMA solution, indicating a promising direction for future research.\n3. Theoretical Foundation: The theoretical analysis providing insights into the convergence behavior of EMA models adds depth and credibility to the proposed method.\n4. Practical Implications: By achieving speedups of up to an order of magnitude or more, LCSC offers tangible benefits in reducing computational resources and training time, which is highly relevant in the context of large-scale model training.", "weaknesses": "1. Search Cost Disparity Between DM and CM: Tables 8 and 9 indicate that the search cost is more substantial for Diffusion Models (DM) compared to Consistency Models (CM), suggesting that LCSC is less effective in reducing training costs for DM.\n2. The coefficients derived by the evolutionary algorithm (EA) lack interpretability, with some being significantly large (\\>6) and others near zero (Fig. 8). Additionally, coefficients learned from different seeds show considerable disagreement, despite yielding similar performance improvements (as seen in Fig. 9). The lack of interpretability may hinder the understanding of the model's behavior and its generalizability. It also raises concerns about the possibility of overfitting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "niaTQ6POuk", "reviewer": "Reviewer_AvHi", "summary": "This paper targets the efficient training problem for diffusion models and consistency models. They find that proper checkpoint merging can significantly improve the training convergence and final performance. Therefore they develop an evolutionary search algorithm for linearly combining the checkpoints. The experiments showcase their LCSC can reduce the training cost and enhance the performance of pre-trained diffusion models.", "questions": "1. How to get such smooth metric landscapes?\n2. How to prove \"a small subset of weights is characterized by coefficients of large magnitude\"?\n3. Why do the models at some iterations have negative coefficients?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The finding that proper checkpoint merging can significantly improve the training convergence and final performance is novel and interesting to me.\n2. The theoretical analyses for EMA are promising and solid.\n3. The experiments are extensive and convincing.", "weaknesses": "1. The details for how to get the metric landscapes are missing. Do you apply a grid search for all $x,y$ with the formula $\\theta_{(x, y)}=\\theta_{n_0}+x\\left(\\theta_{n_1}-\\theta_{n_0}\\right)+y\\left(\\theta_{n_2}-\\theta_{n_0}\\right)$. If so, what is the grid interval? And what is the interval between $n_0$, $n_1$ and $n_2$? It is quite strange that the optimal points are always located inside of the regions surrounded by $n_0$, $n_1$, and $n_2$. Does it mean the models are always oscillating? Please provide more explanation.\n2. The claim in lines 528-529 that \"a small subset of weights is characterized by coefficients of large magnitude\" is interesting but needs more evidence. \n3. Please provide more insights about why the coefficients of some iterations in Figure 5 are negative. It is clear that the negative coefficients mean the models at these iterations have a negative effect on the final performance but why does this happen? Is it because the models at these iterations have a relatively lower FID?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)