Simple and Fast Distillation of Diffusion Models
Authors: Zhenyu Zhou
Defang Chen
Can Wang
Chun Chen
Siwei Lyu
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion models
fast distillation
fast sampling
Primary area: diffusion_based_models
TL;DR: A simple and fast distillation of diffusion models that accelerates the fine-tuning up to 1000 times while performing high-quality image generation.

Abstract: Diffusion-based generative models have demonstrated their powerful performance across various tasks, but this comes at a cost of the slow sampling speed. To achieve both efficient and high-quality synthesis, various distillation-based accelerated sampling methods have been developed recently. However, they generally require time-consuming fine tuning with elaborate designs to achieve satisfactory performance in a specific number of function evaluation (NFE), making them difficult to employ in practice. To address this issue, we propose **S**imple and **F**ast **D**istillation (SFD) of diffusion models, which simplifies the paradigm used in existing methods and largely shortens their fine-tuning time up to $1000\times$. We begin with a vanilla distillation-based sampling method and boost its performance to state of the art by identifying and addressing several small yet vital factors affecting the synthesis efficiency and quality. Our method can also achieve sampling with variable NFEs using a single distilled model. Extensive experiments demonstrate that SFD strikes a good balance between the sample quality and fine-tuning costs in few-step image generation task. For example, SFD achieves 4.53 FID (NFE=2) on CIFAR-10 with only **0.64 hours** of fine-tuning on a single NVIDIA A100 GPU.

Simple and Fast Distillation of Diffusion Models
Zhenyu !.o Defang Chen Can Wang Chun Chen 1.2
Siwei Lyu²
ZZejiang University, State Key Laboratory of Blockchain and Data Security Hangghou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security University at Buffalo, State University of New York {zhyzhou, defchern)@zju. edu. cn
Abstract
Diffusion-based generative models have demonstrated their powerful performance across various tasks, but this comes at a cost of the slow sampling speed. To achieve both efficient and high-quality synthesis, various distillation-based accelerated sam- pling methods have been developed recently. However, they generally require time-consuming fine tuning with elaborate designs to achieve satisfactory perfor- mance in a specific number of function evaluation (NFE), making them difficult to employ in practice. To address this issue, propose Simple and Fast Distillation (SFD) of diffusion models, which simplifies the paradigm used in existing methods and largely shortens their fine-tuning time up to 1000 We begin with vanilla distillation-baal based sampling method and boost its performance to state of the art by identifying and addressing several small yet vital factors affecting the synthesis efficiency and quality. Our method can also achieve sampling with variable NFEs using& single distilled model. Extensive experiments demonstrate that SFD strikes good balance between the sample quality and fine-tuning costs in few-step image generation task. For example, SFD achieves 4.53 FID (NFE=2) on CIFAR-10 with only 0.64 hours of fine-tuning on single NVIDIA A100 GPU. Our code is available at hhtps:: :/op::://itt //////gthhuu// com/ zju pi/ diff sampler
1 Introduction
25
(RFES) INFES)
BDN
DPMe (Aniv22)
OCIR(1)
PO
Guided PD
0C1R21)
(CVPR/23)
UniPC
(heuriP5233)
CD
(ICM223)
Diffusion models have attracted increasing inter- est in recent years due to their remarkable gener- ative abilities across various domains, including image [1 44 (2), video [14] 2), audio 20 24]
Solver -based
PNDM
nrs
IILL:23)
(C1871)
CTM
CTM wo GAN
AMEU
SinD (oura)
ICC
SiDDiv Cours)
15
INFES)
Ours
Distillation-based
and molecular structures (54 These models
INFE4)
INFE4)
(NFE11 (NFE1)
progressively transform a noisy input into re- alistic output through iterative denoising steps. Diffusion models are preferred over other gen- erative models 9 [9] for their high-quality syn- thesis, stable training and strong theoretical foundation rooted in stochastic differential equa- tions [5]]. However, achieving high-quality syn- thesis with diffusion models typically requires hundreds to thousands of sampling steps, result- ing in slow sampling speeds and a significant challenge for practical applications.
(NFE2)
(MF4)
(NFE2)
(NFE4)
INFE
INFEZ (NFE2)
(INEL
(NFE1)X
(NF
10a
10a
10a
10a
Fire- lunentuning time (A100 hours)
Figure 1: Comparison of acceleration methods on dif- fusion models. For better visualization, the time axis is shifted adding one hour to the actual time required. Our method achieves good performance with small fine-tuning cost. Note that it takes about 200 hours to train diffusion model from scratch in this setting.
'Corresponding author. Work partially done during Defang's 's time Zhejiang University.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

4 stops
DDIM steps
SFD-v (ours) 4 steps
Figure 2: Comparison of synthesized images by Stable Diffusion v1.5 41 with guidance scale 7.5.
Recent years have witnessed significant progress in accelerating the sampling of diffusion models [48
30 58 60)
g 38 [S] [1 45 34 50
17 (). These methods typically fall into two categories:
solver-based methods and distillation baaseeaaio--baeddd methods. Solver- based methods [48] 30
58
16
60 8
consider sampling from diffusion models as solving differential equations, and employ fast numerical solvers to accelerate high-quality synthesis. However, these methods are limited by inherent truncation errors, and the sample quality becomes degraded when the number of function evaluations (NFEs) relatively small (e.g., NFE < 5). Distillation bassdd methods, on the other hand, retain the structure of the original (teacher) diffusion model but aim to create a simplified (student) model that streamlines the iterative refinement process diffusion models [3] 45 34 [7 [2 7
38
Extreme intlllatioo-baaele methods even establish a direct one-to- one mapping between the implicit 56] Although distillation-
data distribution and pre-specified noise distribution [1 based methods have demonstrated impressive results, often outperforming solver-based methods in sampling quality given the total NFE budge less than 5, they require expensive computational resources to fine tuning pre-trained diffusion models. As illustrated in Figure the necessary time generally exceeds one hundred GPU hours, even reaching the same order of magnitude required for truining diffasion model from scratch. We attribute the time-consuming fine-tuning process to the following two factors:
10
The mismatch between fine-tuning and sampling steps. There often exists significant fine- tuning costs in existing distillation-base methods that does not effectively contribute the final performance due to the step mismatch. For example, progressive distillation 45 34
fine-tunes the diffusion model thousands of timestamps but only few steps (e.g.. s or fewer) are used in sampling. Besides, consistency- based distillation [50] spends most fine-tuning efforts to ensure the consistency property (5), yet only or steps are used in sampling. Such inconsistencies waste excessive efforts in the fine-tuning process.
The complex optimization objectives. The optimization objectives of distillation-bacse methods are getting increasingly complex, including the of LPIPS (59) 50 17 [6).
adversarial training [46] [7 as well as various regularization terms 17 56). Despite the improved results, these additional components complicate the fine-tuning process.
In this paper, we introduce Simple and Fast Distillation (SFD) of diffusion models, which aims to achieve fast and high-quality synthesis with diffusion models in a few sampling steps, at minimal fine-tuning cost. Starting from the general framework behind distillation- -baed methods, we address the issue of step mismatch by fine-tuning only small number of timestamps that will be used in sampling, which significantly improves the fine-tuning efficiency. The effectiveness of this strategy is underpinned by the key observation that, fine-tuning at specific timestamp can positively impact the gradient direction at other timestamps (Section 3.I. Our SFD is then introduced as a simplified paradigm for the distillation of diffusion models, where the student learns to mimic the teacher's s sampling trajectory while minimizing accumulated errors. We release the potential of this simple framework by identifying and addressing several small yet vital factors affecting the performance (Section 3.2) Moreover, we propose a variable-NFF version of our method named SFD-v, which enables single distilled model to achieve sampling with various steps by introducing a step-condition into the model (Section 3.3
With 2 NFE, our SFD achieves FID of 4.53 on CIFAR-10 21 with training cost of just 0.64 hours on a single NVIDIA A100 GPU, which is 1000x faster than consistency distillation requiring about 1 1156 hours (see Figure I.. Quantitative and qualitative results on additional datasets, including

---

ImageNet 64 [43, Bedroom 256x256 [57] and image generation with Stable Diffusion 41 demonstrate the effectiveness and efficiency of our methods.
2 Preliminary
2.1 Diffusion Models
Diffusion models bridge the implicit data distribution pa and Gaussian distribution ps, by progres- sively adding white Gaussian noise to the data and then iteratively reconstructing the original data from pure noise. Diffusion models are grounded in theoretical framework based on stochastic differential equations (SDEs) [5]] with the forward process injecting noise to data:
dx f(x, t)dt + g(t)dw
(I)
where f(:,t) TRd Rd, g(-) R RR. are drift and diffusion coefficients, and W. TRd denotes the Wiener process [37]. The backward process reconstructs the original data from the noisy input, which can be achieved with a reverse-time SDE that shares the same marginals determined by the forward SDE, i.c., dx = f(x, t) g²( x log (x)]dt + g())d,,, where Vx <oggpeXx Px(x) is known as score
function [13 33 The reverse-time SDE can be further simplified the probability flow ordinary [f(x, t) 2 (t) log Pi (x))dt, In particular, consider f(x, t) 0 and (t) 22 in this paper, i.e.,
differential equation (PF-ODE) [51 16, 3), dx-
dx -V log px)dtt
The score function is estimated Vx (x)
[x, t)/t with noise-prediction model (x f) which is obtained by minimizing a regression loss with the weighing function X(t) for each t 22 48 63
()) X(1)Bx x + te, E
(3)
With the noise-prediction model in place of the score function, the PF-ODE can be written follows
dx
cx(x, t)dt.
Compared to the general reverse-time SDEs, the PF-ODE is preferred in practice for its conceptual simplicity and efficient sampling [51 3) To sample from diffusion model with N steps, one first
draws X Pr NO, m and then numerically solves Eq. by a solver-based method [48]
29
30 58 63
following a hand-crafted time schedule I(NN- = =To = tmin L, ,.N bmax).
The obtained sample sequence {x, I is called the sampling trajectory.
2.2
Distillation- Diffusion Sampling
Through the lens of Eq. we can interpret the poise-predictio model as a gradient field evolves over time, guiding samples towards the data distribution's 's manifold. Solver-based sampling methods
do not change the gradient field and are convenient to implement 48 29 30
58
63 4. However,
discretization errors prevent these methods from generating high-quality images within few sampling
steps. Distillation- based methods address this issue by fine-tuning the gradient field with the reference signals provided by teacher (mostly based method) build "shorteuts" on the sampling trajectory [45] 50 34 17). This basic framework behind distillation-based methods, which we call Trajectory Distillation, is illustrated in Algorithm [ Specifically, starting from latent encodings x 1) with sampled n, the sampling process is written as:
and x (0 S n <
Teacher Xx Colver(xn+1) tn41, tn, K)0),
(5)
Student x Euler(Xx+11 fn+1, En. 1;4) Xn+1 (tx+1 tx fn41),
(6)
where K is the number of teacher sampling steps taken from fn+1 tnp and v. O are the parameters of the student and teacher model, respectively. In each training iteration, the student model & updated with the calculated loss function L(w) d(x); X.,) using a distance metric d(-,-). -). Solver(.,
can be any solver-based method with the fixed 0 to provide reference signals. For example, in progressive distillation [5 34 it is defined the Euler sampler
48] with K 2, while in consistency distillation [50 the Heun sampler [16], = 1, and a consistency loss are used.
Distillation-based methods have demonstrated impressive results but generally incur significant computational overhead. In the following sections, we revisit the trajectory distillation framework and unlock its potential through comprehensive assessment of the key factors affecting the performance.
3

---

Algorithm 1 Trajectory Distillation
Algorithm 2 SFD (ours)
repeat
repeat
Sample Xo from the dataset
Sample XN NN NN I3-1)
Sample n ~ (1, N -11)
for n = - to do
Sample x N (x0:
x Fuuee(xx+11> Int1, ln 1; 1;16)
x Euler (Xner((xn+1, bnt1, En 1; 1)) x, f Solver (x++1 tn11, tn K;
Xn + Solver Solver(Xx*11 tnt1, K; 0)
s < v v/ dx(x xn)
[(t) + d(x;) xn)
X, + detach(x;))
s + v 77 (16))
end for until convergence
until convergence
3 Method
3.1
Smooth Modification of the Gradient Field
As mentioned in Section existing distillation-basee meth- ods incur significant fine-tuning costs that may not effectively
1.
Beseine NOD
contribute to the final sample quality [45] 34 50]. which is a
NODELNe))
key factor overburdening computational resourees. Instead, propose to fine-tune only a few timestamps that will be used in sampling. To validate our strategy, we initialize four different student models (denoted as MODEL (t),), 0 < n S N 1) from pre-trained teacher model using the second-order DPM-Solver(2S) 29 with N = 4 and K = 3. We then fine- tune each MODEL(*s) only on certain timestamp bn+1 1 and make it align with the teacher predictions at the next times- tamp bn- After fine-tuning, we evaluate the performance of each MODEL(1),) at all timestamps by comparing with the teacher's 's sampling trajectory
MODELNe
6. 0.002
4-00170
4-2.516
17.67.53
Figure 3: MODEL( is trained to match the teacher's sampling trajec- tory
but enhance the match- ing at untrained timestamps. The time schedule follows the polynomial sched- ule with = 7, 0.002, t4 = 80.
8 under the same setting. Specifically, we calculate the L2 distance in the following two formulas for all 0 S n, S N
Bascline: : ||x, Euler fn n. 1:0)
()))L( ||x, Euler(Xx,+3 4111, 4n 1; 1:4b%)122.
(8)
and average the results over 1,000 trajectories. As shown Figure the distance calculated using the fine-tuned models is almost consistently smaller than that of the baseline. This is remarkable since each (ODEL( ) only trained to match the teacher's sampling trajectory at a specific 4 Yet, its performance other timestamps is mostly improved, even though different timestamps far apart. This indicates that trajectory distillation does not disrupt the gradient field but enhances smoothly. Since fine-tuning at different timestamps mutually reinforces the model, fine-tuning a fine-grained time schedule is unnecessary. This insight forms the basis our strategy. Beyond efficiency, will demonstrate that our approach achieves high performance in the sequel.
3.2 Simple and Fast Distillation of Diffuusion Models
As for solver-based methods, the cost of a single sampling step varies depending on the design, which is commonly measured by the number of function evaluations (NFE). For the DDIM sampler [48] and other higher-order methods such as DPM-Solver++a step corresponds to one NFE, while two NFEs are required for the Heun sampler [16] and DPM-
30 and UniPC [60]. one sampling
Solver(2S) [29]. In the following, we distill a diffusion model to achieve sampling with two NFEs (N by default). We configure reasonable baseline the CIFAR10 10 dataset (21) and gradually improve the performance through extensive experiments. The improved configuration is proven to be effective across different NFEs and datasets.
Default configuration. The Heun sampler used to generate the teacher sampling trajectory instead of DDIM for efficiency, which has been demonstrated in training consistency models [50 17 We set

---

80
Vantia, Houn
Vanila, DFMI2S!
Kee - 0,000
lee - 0,002
LMTP
L2
70
SFD, Heun
Kea in 0,002, A15
Plotudo Huber
60
SFD, DPM1251
SPD, DPM 18M0
Fea m 0,008, AT5
50
40
30
20
10
3
100
200
300
400
500
500
700
100
200
300
400
500
600
700
9
100
200
300
400
500
600
700
Tairing Earations
Taaning Rerations
Training iter allonn
(a) Teacher solver.
(b) Timestamps.
(c) Loss metric.
Figure 4: Ablation studies of 2-NFE distillation on CIFARIO. The FID is evaluated by 50,000 generated samples with the same latent encodings and is reported every 10 iterations. We achieve the best performance with SFD, DPM-Solvvn+++(3M) teacher, AFS, min = 0.006 and L.1 loss.
K = for Heun sampler, which gives sampling trajectories with 12 NFEs. For the time schedule, not otherwise specified, use the polynomial schedule where p 7, minn 0.002 and fmax 80, following the EDM implementation [16] 50]. The squared L2 loss is used by default. For experiments in this section, we use batch size of 128. learning rate of 5e-5 and fine-tune with 100,000 teacher sampling trajectories generated (around 780 training iterations).
From local to global. We start by analyzing the potential defeets of trajectory distillation. As shown in Algorithm [ trajectory distillation performs local fine-tuning, The term "local" indicates that the teacher model only generates part of the sampling trajectory (i.e., from f,+1 to tz), and the optimization is independent across different n. This raises two defects that limit both efficiency and performance: (i) Higher-order multi-step solvers that require history evaluation records unsupported. (ii) The student model is trained to imitate only part of the teacher's sampling trajectory. During sampling, the errors accumulate since the student is never trained to perfectly fix them. To address these issues, we view trajectory distillation from a global perspective and introduce our Simple and Fast Distillation of diffusion models (SFD) in Algorithm 2 In each training iteration of SFD, we first generate the whole teacher sampling trajectory and let the student imitate it step by step. During this process, the student model generates its own trajectories, enabling to learn to fix the accumulated errors. In Figure 4a| we compare both strategies (marked as "Vanilla" and "SFD") using the default configuration. we focus on SFD and seek to release its potential for efficient distillation of diffusion models.
is shown that SFD exhibits better performance. In the following sections,
Efficient solver. One of the key components that affects the efficiency of distillation-base basedd methods is the choice of the teacher solver. To compare the performance of different solvers, we conduct experiments on both trajectory distillation and SFD with 3 representative solver-based methods: second-order Heun sampler, second-order DPM-Sovler(2S) and third-order DPM-Solver++<3M) Since history evaluations are unavailable, we exclude DPM-Solver+++ for trajectory distillation. The NFE of teacher sampling trajectories is kept 12 consistently (K = 6 is hence used for DPM- (3M) and the results are shown in Figure 4al (3M-S)llvr+++(MM) stands out, and the Heun sampler is shown to be suboptimal. Therefore, for distillation-basee methods with trajectory distillation involved, it recommended to explore replacing the Heun sampler (or DDIM sampler) with DPM-SS)ver(2S). We leave this to future works.
Minimum and maximum timestamps. Choosing DPM-Solver++ the teacher solver, we improve SFD by adjusting the start and end timestamps during training and sampling. For the minimum timestamp mmin. empirically find that slight increase improves the student and teacher sampling performance across various datasets. An ablation study of /min on CIFAR 10 dataset is shown in Figure 5] We increase fmii from 0.002 to 0.006. This change provides consistent improvements across different pre-trained diffusion models. For the maximum timestamp, we introduce analytical first step (AFS) 6 663 4 in the generation of student sampling trajectories, which takes one estimated step (xN, (N) x XN /s/v/11 I fy at the beginning of sampling to save one NFE. Therefore, to
obtain 2-NFE SED with AFS applied, we use N = 3 and K = 4. As shown in Figure 4b] using AFS largely boosts the performance of SFD, indicating that one more inaccurate step can outperform one less step. This improvement also benefits from the nature of SFD, where the emror incurred in the first step can be fixed by later steps (see the visualization in Figure The detailed algorithm of SFD with AFS is included in Appendix C
5

---

PC2
Gax- 0.002
Teacher trajectory
0.004
373.
Student trajectory
10
573, M775
Gen 0.030
4.0
3.5
SFD- NFG2
-2
SPOw NrcL
2.0
MFE
1000
2000
3000
4000
15
PC1
Figure 5: Ablation study on Figure 6: Extrapolation ability Figure 7: Visualization the effec- Eiin with (3M++ on untrained NFE.
tiveness of SFD.
various distance metrics for the loss function including squared 1.2 distance, L.1 distance, LPIPS distance [59] and Pseudo-Huber distance [49]. Among these metries, I.1 distance outperforms. Note that the LPIPS distance is trained evaluate the perceptual distance between two images but not corrupted ones, which may explain its suboptimal performance.
Loss metric. In Figure 4c
With these improvements, the SFD achieves fast convergence with only around 300 training iterations, which only takes around minutes on a single NVIDIA A100 GPU. The perfor-
Table Quantitative results of the ablations.
Method
Teacher
Gmin
AFS
Loss
FID
Vanilla
Heun
0.002
NIA
1.2
46.84
mance of the obtained 2-NFE SFD is even com-
Vanilla
DPM(2S)
0.002
N/A
L2
16.69
parable with the 2-NFE model trained by pro- which takes more
SED
Heun
0.002
Fabse
L2
20.88
gressive distillation 45 34
DPM(2S)
0.002
False
L2
12.50
than 100 hours under our estimation. The quan- titative results are included in Table
DPM++(3)
DPM++(3
SFD
DPM++(3M)
0.002
True
L2
7.17
To verify our findings in Section 3.I we test the extrapolation ability of SFD with untrained NFEs on CIFAR1O. The results are shown in Figure 6 where the markers indicate the NFEs our methods are trained to sample with. Take "SED, NFE4" as an example, where the SED is only trained to sample with NFE of 4 and its performance is marked by to sample with untrained NFEs (i.e., 2,3,5,6), even though the timestamps have never been trained in these cases, the performance is still decent and largely outperforms the DDIM sampler (DDIM with NFE of gives FID of 35.62, far exceeds the range of the figure). This empirically verifies our hypothesis that the gradient field is not disrupted but is smoothly enhanced. The change of the gradient field of certain timestamp can also change that of nearby timestamps in a similar way.
SFD
(PM++(3M)
0.006
True
L22
5.67
SED
DPM++(3M
0.006
True
LPIPS
5.10
SFD
DPM++(3M)
0.006
True
PH
4.90
SED
DPM++(3)
0.006
True
L1
4.57
When using this SFD
projection technique proposed in [ to visualize the sampling trajectories generated by SFD with NFEs and that the teacher solver SFD is trained to imitate. Due to the use of AFS, the first sampling step of SFD is inaccurate. But the accumulated errors are largely reduced in the following steps thanks to the global distillation used in We include more visualised trajectories in Appendix D.2
Moreover, in Figure 7 we leverage the
SFD discussed in Section 3.2
3.3 Towards Variable-NFE Distillation
One attractive property of diffusion models is that the sample qual- ity can be consistently improved by increasing the sampling steps, which is currently unsupported by most distillation-basse methods. Progressive distillation [45] 34 partially addresses this issue using the multi-stage training. However, the model saved in each training round only supports sampling with a certain trained step (i.e.. 1, 2, 4, 8, ...). The sample quality of consistency models [50] 49 designed for the one-step sampling also deteriorates under larger sampling steps revealed by [17]. Moreover, the unique encoding property of ODE-based diffusion sampling is corrupted in multi-step consistency models due to the noise injected in every step.
10
15
20
Taining 1to Daaeeeenn (x1000)
Figure 8: Ablation study on the type of condition.

---

00
20
20
10
P12
P22
122
-10
-10
-1
1200
1500
10
500
-2g
1000
1500
20
10
1000
-gg
1500
30
20
(a) Guidance scale 1.0.
Guidance scale 4.0.
(c) Guidance scale 7.5.
Figure 9: We visualize 20 sampling trajectories generated by DPM-Solver++(2M) 30 with 20 steps using the three-dimensional projection technique proposed in +
To address this issue, consisteney trajectory models (CTM) introduce a new condition into the student model, which specifies the next time to arrive, referred to as the conditin. Unlike CTM, introducing step-condition to our SFD is more efficient. By informing the student model of the number of sampling steps, our SFD can perform sampling with different NFEs. We refer to this variable-NFF. version of our method as SFD-v.
In every training iteration of SFD-v, the total number of sampling steps N is first sampled uniformly from pre-specified list of steps. Then, the time schedule I((N) ) is generated, and the subsequent training process is the same as training SFD. As shown in the ablation study in Figure g the step- condition consistently outperforms the tonzi-conditonn For the injection of step-condition, we treat the same way as the time embedding in diffusion models (see Appendix D.I for more details). We include the algorithm of training SFD-v in Appendix C
3.4 Distillation under Classifier-free Guidance
Stable Diffusion [41] latent diffusion model combined with classifier-free guidance [13], has shown to be highly effective in high-resolution image generation. The classifier free guidance extends the flexibility of the generation of diffusion models by introducing the guidance scale Given conditioning information the presccpredintion model is rewritten as
(x, t, c) = weo(x, t, + wxx
However, Stable Diffusion requires a large number of network parameters and sampling steps to produce satísfying generation. Moreover, the cost of every sampling step doubles since both conditional and unconditional evaluations are involved in Eq. 9 Distilling the Stable Diffusion model into a few steps is challenging because source-intensive requirements and the flexibility given by the guidance scale. To address this issue, existing methods either introduce an w-condition into their model 34
or simply discard the guidance scale 155 46
Here, we propose a new strategy with the observation on the sampling trajectories generated by Stable Diffusion under different guidance scales. Following the three-dimensional projection technique proposed in [4, we visualize the sampling trajectories generated by Stable Diffusion in its latent space using DPM-Solver++H2M starting from 20 fixed latent encodings. As shown in Figure 9 sampling trajectories projected to the three-dimensional subspace exhibit a regular boomerang shape, which consistent with the findings in the previous work (4]. Furthermore. trajectories become more complex as the guidance scale increases, making trajectory distillation on the high guidance scale even more challenging. This observation naturally leads our strategy: perform distillation with a guidance scale of and sampling with any guidance scale. Our strategy enables accelerated training since the unconditional evaluation in Eq. O eliminated.
observe that the sampling
Experiments
4.1 Experiment Setting
Pre-trained models and datasets. Both the network parameters of student and teacher models are initialized from pre-trained diffusion models provided by EDM [16] and LDM [4T]. We report quantitative as well as qualitative results on datasets with various resolutions including CIFAR10
7

---

32x32 [2 ImageNet 64 [43] and satent-space ASSUN-Bedrood 256x256 [57]. For Stable Diffusion we use the v1.5 checkpoint and generate images with resolution of 512x512.
Training. The configuration obtained in Section 3.2 can be applied to different NFEs and datasets. Generally, in the training of SFD and SFD-v, we use DPM-Solver+++ (D) the teacher solver with K = 4 (see Appendix D.2 for an ablation study on K). The use of adjusted Emin 0.006, AFS and LI loss introduced in Section 3.2 all lead improved results. Minor changes are needed for texi-to-image generation with Stable Diffusion, where we use DPM-Solver++ which is the default setting used in Stable Diffusion and K = 3. In this case, Emin increased from 0.03 0.1 and the AFS is disabled due to the complex trajectory shown in Figure 9 These experiment settings are collected in Table o Appendix.
Optimization. We use Adam optimizer across all datasets. A learning rate of 1e-5 is used for ImageNet and LSUN-Bedroom while 5e-5 is used in other cases. We divide the learning rate by 10 halfway through training. Our SFD trained with total of 200K teacher trajectories generated (around 1.5K training iterations). We train SFD-v to enable sampling with NFE from 2 to 5, the total training iterations is multiplied by 4 accordingly. All experiments are conducted with a maximum of 4 NVIDIA A100 GPUs. To enable a batch size of 128 using Stable Diffusion, we accumulate the gradient through several rounds.
18
with 31 = 0.9 and B2 = 0.999 and a batch size of 128
Evaluation. We measure the sample quality via Fréchet Inception Distance (FID) [ with 50K images in general. For text-to-image generation, we use a guidance scale 7.5 to generate 5K images with prompts from the MS-COCO (23) validation set. The FIDD is evaluated following the protocol in 28 34 46) where the validation set serves as reference images. The CLIP score computed using
the ViT-g- CLIP model 40) trained on LAION-2B 47
Table 2: Results on CIFARIO 32 x 32.
Table 3: Results on ImageNet 64 x 64.
Training time (A100 bours)
Training time (A100 bours)
Method
NFB
NFE
FID
Solver-based Methods
Salver-based Methods
DDIM 48
10
15.69
0
DDMM AS
10
16.72
50
2.91
0
50
4.09
(PM++(3) SV
5
24.97
0
(PM++3M) 3I
25.49
10
3.00
0
5.67
AMED-Plugin
5
6.61
0,08
AMED AMED-Plugin
13.83
0.18
10
2.48
0.11
10
5.01
0.32
GITS
4
10.79
10
2.49
0.01
10
4.48
0.02
Diffusion Distillation
Diffussin Distillation
PD
45
9.12
195
PD 45
15.39
5533
2
4.51
171
2
8.95
4611
Guided PD
34
1
8.34
146
Guided PD
34
22.74
5533
2
4.48
~ 128
9.75
< 4611
4
3.18
119
4.14
4150
50
3.55
1156
6.20
< 7867
2.43
1156
4.70
7867
CTM
17
1.98
CTM F
2.06
902
CTM 10
w/o GAN loss
>
60
1.90
902
SFD (ours) (second stage)
5.83
4.88
SFD (ours) (secand stage)
12.89
6.86
SFD (ours)
4.53
0.64
SFD (ours)
10.25
3.34
3.58
0.92
6.35
4,63
3.24
1.17
4.99
5.98
3,06 5
1.42
4.33
7.11
SFD-v (ours)
2
4.28
SFD-v (ours)
9.47
3
3.50
5.78
3.18
4.26
4.72
23.62
5
2.95
5
4.21
4.2
Main Results
We mainly compare our proposed SFD and SFD-v with progressive distillation 45 and consis- tency distillation [50 I7]. Table and we report unconditional and conditional results on the pixel-space image generation. To compare the training cost, we estimate the training time measured by hours consumed on single NVIDIA A 100 GPU, following the training settings in the original papers. The detail of our estimation is included in Appendix B Our SFD achieves comparable results as progressive distillation but only requires a very small fine-tuning cost (100x to 200x speedup). At the same time, it is hard for solver-based methods to give high-quality generation within a few steps
8

---

due to increased errors. In accordance with our finding in Section 3.1 the SFD-v shows consistently better results than SFD, although the training cost of SFD-v and SFD roughly the same for each specified sampling step. These observations also apply to the performance of SFD and SFD-v on the latent-space image generation on
we show the performance of our methods in terms of FID and CLIP scores with guidance scale of 7.5. The results demonstrate the effectiveness of our strategy proposed in Section 3.4 where the training is performed with the guidance scale set to 1. The qualitative results are shown in Figure 2 We include
shown in Table In Table 5
more results in Appendix D.2
Although generating images with one NFE is possible with SFD and SFD-v, we find suboptimal. To address this, we propose a second-stage one-NFE distillation, initializing network parameters from a fine-tuned SFD model. In this second stage, the teacher solver is set to DDIM (as used in SFD), and we use AFS (N = 2) and K The training procedure remains the same as that of SFD. The results, marked as "second-stage", are reported in Tables to We provide an ablation study on the effectiveness of the second stage and the LPIPS metric [59] in Figure TO] The second-stage training significantly boosts performance and is more efficient. Additionally, combining L1 loss with LPIPS loss yields better results. Since the teacher requires a smaller NFE, the training of each iteration of the second-stage distillation fast. Therefore, for CIFAR 10/mmaeeNet we perform second-stug -ttog distillation with 2000/800K sampling trajectories (around I 15/6K training iterations), and the learning rate is set to 10 times larger. For LSUNN-Bedroon use 800K trajectories and disable the LPIPS loss.
SPALLL
13
75
150
Iterations
Figure 10: Ablation study on one-NFE distillation.
Table 4: Results on LSUN-Bedroom 256 x 256. Table 5: Text-to-image generation with Stable
Method
NFE
FID
Diffusion v1.5 4T. *; Reeported in the original paper (34. We use a guidance scale of 7.5, which
(3M)+ 30
8
4.61
AMED Plugin 63
8
4.19
is the default setting the original repository.
PD 45
1
16.92
2
8.47
Method
Steps
FID-5K
CLIP Score
CD
1
7,80
D2M+
20
2
91.5 (98 8
0.20 (0.19*)
2
5,22
4
31.1 (34.1 1*)
0.29 (0.29*)
SED (ours) (sccond-stage)
1
13.88
8
25.1 (25 6*)
0.32 (0.30*)
SED (ours)
2
10.39
Guided PD
2
37.3
0.27
6.42
4
26.0
0.30
4
5,26
g
26.9
0,30
SmppFusion SFD (ours)
22
B
24.2
0.30
SFDay (ours)
2 9.25
42.9
0,24
5.36
27.6
0.27
4
4.63
24.2
0,28
5
23.5
0.29
Conclusion
In this paper, we introduce Simple and Fast Distillation (SFD) of diffusion models to achieve fast and high-quality generation with diffusion models in few sampling steps at minimal fine-ture fine-tuning cost. Through a comprehensive investigation of several important factors, we unlock SFD's potential, achieving sample quality comparable to progressive distillation while reducing fine-tuning costs by over 100 times. To enable sampling with variable NFES using a single distilled model, we propose SFD-v, which incorporates step-condition as additional input. Our methods strike good balance between sample quality and fine-tuning costs for few-step image generation, offering a new paradigm for distillation-baat accelerated sampling of diffusion models.
Limitation and future work. Despite demonstrating efficient training, the FID results of our methods currently do not match those of the Mfae--ofthhe-rt In future work, we plan to further explore the core mechanisms affecting the performance of trajectory distillation. Additionally, given the recent discoveries of the remarkable regular geometric structure of diffusion models [4], aim to tailor an appropriate time schedule for our methods. We also intend to validate the effectiveness of the factors we have discussed in other distillation-bbseed methods.
Broader impacts. Similar to existing works on content creation, our methods have the potential to be misused for malicious generation, which could have harmful social impacts. However, this risk

---

can be mitigated through advanced deepfake detection techniques. By continuously improving these detection methods, we can help ensure the responsible and ethical use of our technology.
6 Acknowledgement
Zhenyu Zhou and Can Wang are supported by the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study, China (Grant No: SN-ZJU-SIAS-ON National Natural Science Foundation of China (Grant No: U U1866602) and the advanced computing resources provided by the Supercomputing Center of Hangzhou City University.
References
[1] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbot, and Eric Gu. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv:2303. 042488, 2023.
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[3] Defang Chen, Zhenyu Zhou, Jian-Ping Mei, Chunhua Shen, Chun Chen, and Can Wang. geometric perspective on diffusion models. arXiv preprint 2rXiv:2305.99947, 2023.
[4] Defang Chen. Zhenyu Zhou, Can Wang, Chunhua Shen, and Siwei Lyu. On the trajectory regularity of ODE-based diffusion sampling. In International Conference on Machine Learning. pages 7905-7934. PMLR, 2024.
[5] Giannis Daras, Yuval Dagan, Alexandros G Dimakis, and Constantinos Daskalakis. Consistent diffusion models: Mitigating sampling drift by learning to be consistent. arXiv preprint arXiv:2302.09057 2023.
[6] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie:
Higher-order denoising diffusion solvers. In Advances in Neural Information Processing Systems, 2022.
[7] Weilun Feng, Chuanguang Yang, Zhulin An, Libo Huang, Boyu Diao, Fei Wang. and Yongjun Xu. Relational diffusion distillation for efficient image generation. In ACM Multimedia, 2024.
[8] Martin Gonzalez, Nelson Fernandez Pinto, Thuy Tran, Hatem Hajri, Nader Masmoudi, et al. Seeds: Exponential sde solvers for fast high-quality sampling from diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[10] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Josh Susskind. Boot: Data-free distillation of denoising diffusion models with bootstrapping. arXiv preprint arXiv:2306.00554, 2023.
[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6633 2017.
[12] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint urXiv:2207.12594 2022.
[14] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
10

---

[15] Aapo Hyvärinen. Estimation of non-normalizze statistical models by score matching. Journal of Machine Learning Research, 6:695-709, 2005.
[16] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
[17] Dongjun Kim, Chich-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023.
[18] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980 2014.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[20] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
[21] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
[22] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. Advances in Neural Information Procexsing Systems, 36, 2024.
[23] Tsung-Yl Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[24] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-ann fext-to-audio generation with latent diffusion models. arXiv preprint 2rXiv:2301.125000 2023.
[25] Liyuan Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 03265,, 2019.
[26] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022.
[27] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
(28) Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high-quality diffusion-based text-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
[29] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, 2022.
[30] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arX1v:2211.01095, 2022.
[31] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388 2021.
[32] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint 10..0:2310.04378 2023.
11

---

[33] Siwei Lyu. Interpretation and generalization of score matching. In Proceedings of the Twenty- Fifth Conference on Uncertainty in Artificial Intelligence, pages 359-366, 2009.
[34] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306, 2023.
[35] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International Conference on Machine Learning, pages 7176-7185. PMLR. 2020.
[36] Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step cext-to-image diffusion model with variational score distillation. arXiv preprint arXiv:2312..52.99 2023.
[37] Bernt Oksendal. Stochastic differential equations: introduction with applications. Springer Science Business Media, 2013.
[38] Kushagra Pandey, Maja Rudolph, and Stephan Mandt. Efficient integrators for diffusion generative models. arXiv preprint arXiv:2310.0789, 2023.
[39] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages $748-8763. PMLR, 2021.
[41] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[42] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[43] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-255 2015.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, pages 36479-36499, 2022.
[45] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2022.
[46] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. arXiv preprint arXiv:2311.17042. 2023.
[47] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 222294.222944 2022.
[48] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
[49] Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14188 2023.
12

---

[50] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine learning, 2023.
[51] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[52] Wujie Sun, Defang Chen, Can Wang. Deshi Ye, Yan Feng. and Chun Chen. Accelerating diffu- sion sampling with classifier- based feature distillation. In 2023 IEEE International Conference on Multimedia and Expo (ICME), pages 8 810-815. IEEE, 2023.
[53] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[54] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geo- metric diffusion model for molecular conformation generation. arXiv preprint 2tti::220230092... 2022.
[55] Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingbo Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. arXiv preprint arXiv:2311.09257, 2023.
[56) Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. arXiv preprint XrXiv:231 1882888 2023.
[57] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scall scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506. 03365, 2015.
[58] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In International Conference on Learning Representations, 2023.
[59) Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[60] Wenliang Zhao, Lujia Bai, Yongming Rao. Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04880 2023.
[61] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In International Conference on Machine Learning, pages 22390-42402 PMLR, 2023.
[62] Kaiwen Zheng, Cheng Lu. Jianfei Chen, and Jun Zhu. Dpm--slvvr-v3: Improved diffusion ode solver with empirical model statistics. arXiv preprint arXiv:2310.13268 2023.
[63] Zhenyu Zhou, Defang Chen, Can Wang, and Chun Chen. Fast ode-based sampling for diffusion models in around 5 steps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2024.
13

---

A Related Works
Solver-based methods. As sampling from diffusion models can be interpreted as solving the PF-ODE [51], various kinds of training-free solver-based methods are designed utilizing classical numerical methods including Euler's ' method (DDIM (48),, Heun's 's second method (EDM (16). linear multi-step method (PNDM [26) and iPNDM (58) and predictor-corrector framework (UniPC [60])
Some works focus on the semi-lineed structure of the PF-ODE, approximating its solution by Taylor expansion (DPM-Son) SoMver [29],
[30 and their generalized variant SEEDS [8) and polynomial extrapolation (DEIS (8)). Besides these training-free -fre methods, some works further reduce the discretization error of classical numerical methods by learning intrinsic information with extra computational overhead. GENIE 6 applies the second-order truncated Taylor method and utilizes the gradient of noise-prediction model w.r.t. time distilled from the pre-trained model for accelerated sampling. DPM-Solver-v3 [62] seeks to search for the optimal parameterization for fast sampling using the Empirical Model Statisties (EMS) calculated from pre-trained models. AMED- Solver [63] achieves fast sampling resorting to the mean value theorem validated by the geometric structure of sampling trajectories and trains small network that predicts the optimal intermediate timestamps. GITS [ optimizes the time schedule through a dynamic programming utilizing the trajectory regularity of the diffusion models.
Aside from solver-based methods, distillation- based methods demonstrate their superiority in sam- pling speed and high-quality generation. Current literature can be categorized into three classes.
Trajectory distillation. Trajectory distillation originates from the primary work [I], which proposed the first one-step diffusion model with the idea of knowledge distillation. The basic framework behind is to train student model to imitate the teacher's s sampling trajectory. Progressive distillation (PD)
proposes gradually reducing the sampling steps with a multi-stage strategy. In each training round, the student model is fine-tuned to merge two DDIM steps into one step and serves as the teacher in the next training round. Following PD, trajectory matching at feature space utilizing pre-trained classifier is shown to be effective in RCFD (52) and RDD (7]. Motivated by the idea of operator learning, DSNO 61 presents novel way of distillation by simultaneously predicting the whole sampling trajectory with specially designed temporal convolution blocks.
45 34 [2)
Consistency distillation. Originated from consistency models [50), which is special case of its concurrent consistent diffusion models [5], consistency distillation introduces a new way of distillation where the denoising outputs on the sampling trajectory are kept consistent. Consistency distillation shows remarkable ability in one-step generation and has become popular in Distillation-basee methods. Latent consistency models (32) demonstrate the effectiveness of consistency distillation in latent space. Consistency trajectory models (17) generalize the consistency to any timestamp, enabling an unrestricted traversal on sampling trajectory and producing state-of-the-art results.
Distribution matching. The idea of distribution matching for distilling diffusion models is first introduced in text-to-3D generation [39 as score distillation sampling (SDS) and is later improved to variational score distillation (VSD) (53. Unlike the training of diffusion models, which relies on sample-wise reconstruction, distribution matching discards the coupling of noise-image pairs set by diffusion models and matches the real and reconstructed samples at a distribution level. The is recently validated in image generation.
effectiveness of both SDS [46] and VSD [56]] 36
Our methods fall into the first category but differ from progressive distillation in three ways. (i) We view trajectory distillation from global perspective, where we generate the whole teacher sampling trajectory in each iteration, and let the student model imitate step by step. (ii) We simplify the multi-stage strategy (more than ten stages) into one or two stages. (iii) The proposed SFD-v enables sampling with different NFEs using a single distilled model. Our methods differ from DSNO two folds. (i) Our methods enable the student model to reduce accumulated errors in previous steps, while in DSNO, the intermediate samples on the predicted sampling trajectory are not directly connected. (ii) Our methods slightly change the network architecture, while in DSNO, it is largely redesigned. Besides, the model complexity in DSNO increases as the length of the sampling trajectory increases.
B Estimation of Training Cost
In this section, we illustrate our estimation of training cost on the comparison methods shown
in Table 2 and 3 For solver-based methods like AMED-Solver [63] and GITS (4, we borrow
14

---

the reported training cost in original papers since similar devices are used. Efforts are put on the estimation of distillation-base based methods. Due to limited resources, we do not fully re-implement these methods. Generally, after the training stabilizes, we collect the consumed time spent on several training iterations and then estimate the total training costs. Two NVIDIA A100 GPUs are used in our estimation and we double the estimated training costs to report the final results. For fair comparison, all of these methods are re-implemented with the EDM [16] repository with its provided pre-trained diffusion models. We note that the cost of training a diffusion model on CIFARIO 10 dataset from scratch is around 200 A100 hours.
B.1 Training Costs on CIFARIO
Progressive distillation. Following the setting in the original paper 45 we use a batch size of 128 and strictly follows the original algorithm. Adam optimizer as well as exponential moving average (EMA) is also used. After the training stabilizes, it costs around 140.5 seconds for every 320 training iterations with 2 NVIDIA A100 GPUs. As total of 800K training iterations are required in the original paper, we estimate its training cost by 800 x 1000 x 140.5 x 2/(320 x 3600) = 195.1 A100 hours. As for Guided PD [44, we only report the estimated training cost for its second-stage distillation, which requires a total of 600K training iterations. We roughly use the above statistics and estimate its training cost by 600 x 1000 x 140.5 x 2/(320 x 3600) = 146.4. For the training cost under larger NFE, we scale the estimated training cost accordingly following the total number of training iterations.
Consistency distillation. The consistency distillation 50 uses batch size of 512 and trains for total of 800K training iterations. Following the original paper, we use the LPIPS [59] loss metric and the Rectified Adam optimizer [25]. It costs around 52 seconds for every 20 training iterations with 2 NVIDIA A100 GPUs. The training cost is thus estimated by 800 x 1000 x 52 x 2/(20 x 3600) = 1155.6 A100 hours.
Consistency trajectory models. Following the original paper we use batch size of 256 and train with mixed -exxedprrccisio.. For the first 50K training without the GAN loss, it costs around 21.5 seconds for every 20 training iterations and 38.5 seconds are needed for the later 50K training with GAN loss involved using 2 NVIDIA A100 GPUs. The training cost of a total of 100K iterations is estimated by 50 x 1000 (21.5 + 38.5) x 2/(20 x 3600) = 83.3 A100 hours. If the GAN loss is disabled during training, the training cost will be reduced to 59.7 A100 hours.
B.2 Training Costs on ImageNet
Typically, the training of ImageNet uses large batch size (for example, the original setting is 2048 for all the methods below) which requires extensive resources (usually 64 GPUs are needed). Due to limited resources, we estimate the training cost with batch size of 256 and multiply by 8 which should gives an upper bound of the practical training cost.
Progressive distillation. In progressive distillation [45] well as its guided version 34 total of 600K training iterations are required (50K for 8 rounds and 100K for rounds). With a batch size of 256, it costs around 664 seconds for every 320 training iterations with 2 NVIDIA A100 GPUs. The training cost is thus 8 x 600 x 1000 x 664 x 2/(320 x 3600) = 5533.3 A 100 hours. For the training cost under larger NFE, the training cost is scaled accordingly as done before.
Consistency distillation. Following the original setting [50], mixed srixed-precision optimization is applied. Other settings is similar to that in CIFAR LO illustrated above. With a batch size of 256, it costs around 59 seconds for every 20 training iterations with 2 NVIDIA A 110 GPUs As total of 600K iterations are required, the training cost is estimated by 8 x 600 x 1000 x 59 2/(20 x 3600) 7866.7 7 A100 hours.
Consistency trajectory models. Expect for training iteration of 30K, other settings are consistent as in CIFARIO 1 training. For the first 10K training iterations without GAN loss, it costs around 122 seconds for every 20 training iterations with 2 NVIDIA A100 GPUs. For the later 20K training iterations with GAN loss, 142 seconds are needed. The total training cost is estimated by 8 x 1000 x (10 122 + 20 142) 2/(20 x 3600) = 902.2 A100 hours.
15

---

C
Algorithms
All the algorithms involved in the main text are illustrated below.
Algorithm 3 Trajectory Distillation
Algorithm SFD (our method)
Input: model parameters & 0, learning rate 7, Solver(-,.,.. distance metric .(.,)), num- ber of student sampling steps N, noise sched- ule ftn In number of teacher sampling steps between every two noise levels K, dataset 'D. repeat
Input: model parameters 0, learning rate y), Solver(-, -), distance metric d(--)), num- ber of student sampling steps N, noise sched- ule ff, h number of teacher sampling steps between every two noise levels K.
repeat
Sample X0 DD
Sample XN = NN ~ NN( I
Sample U00 N - 1)
for n = N - 1 to 0 do
Sample Xx+1 ~ NNxoi 11+11)
x < Fxule((n+1::444 Inz 1; v0)
x + Exler(xa+11 tn+1, In, 1:)
X, Solver int1, K;
X, f (iiver(xn++,444
bn K; K;
v < v V &x) &n)
((t) + d(x)) xnn
X, + detach(x;)
sb + 1 - ((66)
end for until convergence
until convergence
Algorithm 5 SFD with AFS
Algorithm SFD-v (variablc-NFE)
model parameters y 0, learning rate v, Solver(-,', ".)) :), distance metric d(::),, num- ber of student sampling steps N, noise sched- ule {tn o number of teacher sampling steps between every two noise levels K. repeat
Input:
Input: model parameters 0, learning rate y, Solver(;;'s :), distance metric d(.,)). )) num- ber of student sampling steps N, noise sched- ule {fn J number of teacher sampling steps between every noise levels K, list L. Initialize: inject step-condition as new input to the model as ev t, c, step = N)
Sample XN NN &0
é - XN/
repeat
XN-1 f XN + (tN-1 - tNjē
Sample N LLL) and generate {t, [
Xy-1 + Solver (NN. t.. tN K;
Sample XN = NN ~ ((; 1)
for N - to do
for n N to do
x < Euler(Xx.41 tn 13) (xn+1 (x,er bnt1sbn K; 0) v <- V/V vi((x) sen)
x t- Fuler(xn+1,, (1.,1,,141166 Eno 1;
X, Solver
Xn + Solver(Xn+1, int1, ta K;
t 3b - 1/V di((xh
X, +- detach(x;))
X, detach(x;))
end for until convergence
end for
until convergence
Algorithm 7 Second one-NFE distillation
Algorithm SFD/SFD-v sampling
Input: = 0, r7. d(.,)), (to, 1,), K. # AFS can be used similar to Algorithm 5 Step-condition can be used as Algorithm repeat
Input: model parameters th, number of student sampling steps N, noise schedule {tn ( Initialize: Sample XN ~ (N
# AFS can be used similar to Algorithm # Step-condition can be used as Algorithm
Sample X1 ~ t0;;
1 + Euler(x1, (1, to 1; 1;)
for = N - to 0 do
Xo < Euler Euler((x,, t1, K; 0)
x, <- Fuler (xterr(xx+10 In41, I,, 1;1)
end for return Xy
e + 2 Vvd((x) xo)
until convergence
16

---

D Additional Results
D.1 Inject Step-condition into the Model
In this section, we illustrate the detail of how to inject the step-condition into diffusion models. We take the EDM [66] models trained on CIFARIO dataset as an example. The modifications on models trained on other datasets are similar.
Generally,
step-condition the same the timestamps input to diffusion models. The step- condition go through similar operations like timestamps to obtain the step-embedding which is then added the time-embedding in every UNetBlock. For CIFARIO model, we use the +DPM+ backbone proposed in [5I] where the positional embedding is applied to encode the input time. The detailed modifications are shown in Algorithm 9 [2 Note that for class-conditional models, do not add the class- embeddinn to the step-embedding which we find to be suboptimal.
Algorithm 9 Original network for CIFARIO
Algorithm 10 Network with siep-condition
Class MeweeNN(((orc..MMouuu def init ():
Class Song Cer.eeettorc....Moouuee def init O
self.map noise PositionalEmbed()
self.map noise PositionalEmmed0)
self.map laae__aerr-- Lincar() self.map
self. hatlmmpp_lyeel Linear(
layerl lay_er
Lincar()
self.map laperllyerrrr_ Lincar(
self.map_stter PositionalEmbed0)
self.step_layer] Linear()
seff.steplaaee Linear()
def forward(x, noise):
def forward(x, noise, step): emb self.map_ noise(noise) if class cais__ooditional:
emb = self.map_noie(noiie noise(noise) if class conditional:
add class embedding to emb emb silu(self.nn map. layerO(emb)) emb silu(self. laye
# add class embedding to emb emb silu(self.map. _ap_layerr(emmb))))
emb self.map_step((tee)) emb self.map
(emb))
emb_s S silu(self. staiseeffstep_lllerOOeeeee emb silu(self. layer lamerl s))
for every Ulocklock: UNetBlock(x, emb)
for every UNetBlock:
x UNetBlock(x, emb, emb_s _)
Algorithm 11 Original UNetBlock for CIFAR10
Algorithm 12 UNetBlock with step-condition
Class ENNeBlockttorch.nn.Moddul def init ():
Class wentBoock(trrch.n.MModd. def init O):
self.affine Linear(
self.affine Linear()
self.affine Linear()
def forward(x, emb):
def forward(x, emb, emb_s):
params self.affin(ceb)
params self. self.affine(emb)
params 5 = self.affine(emb.
params
paramsssss.... params_s
silu(seff.norm()))
x = silu(self.norm(x)
17

---

Table Experiment settings used in the main text. t: Generated teacher sampling trajectories. When training SFD-v, it refers to the average generated trajectories used for each NFE. *; We force a batch size of 128 by accumulating the gradient 8 rounds.
Hyperparameter
CIFARIO
ImageNet
Stable Diftusion
Teacher solver
DPM++(3M)
DPM++(3M)
DPM++(3M)
DPM++(2M)
K
4
Emin
0.006
APS
True
True
True
False
Generated traj-f
200K
200K
200K
100K
Learning rate
5e-5
le-5
1o-5
5c-5
Optimizer
Adam
Adam
Adam
Adam
Loss metric
Batch size
128
128
128
16*
Hixed-Precision
True
True
True
Thue
Number of GPUs
4
4
Table 7: FID results CIFAR-10. The lines with gray background are the results reported in the main text. t: Second-stage one-NFE distillation. For each row of the results of SFD-v, the number of reported results corresponds the the length of the list of sampling steps in Algorithm 6
NFE
Method
3
5
6
8
SFD (100K)
21.14
4.57
3.66
3.26
3.06
2.97
2,87
2.85
SFD (200K)
5.83
4.53
3.58
3.24
3.06
SFD-v (800K)
4,28
3.50
3.18
2.95
SFD-v (800K)
3.58
2.97
2.94
2.87
SFD-v (2000K)
11.35
4.16
3.44
3.11
2.95
SFD-v (2000K, conditional)
9.17
3.45
2.85
2.76
2.63
Table Ablation study on the number of teacher sampling steps on CIFAR10 dataset. We report pairs of FID and fine-tuning time (A100 hours).
NFE
2
17
18.61/0.22
6.51/0.28
4.57/0.32
4.99/0.38
5.66/0.42
40.15/0.26
7.76/0.32
3.60/0.39
3.66/0.46
3.84/0.53
3.97/0.59
25.96/0.33
5,21/0.42
3.34/0.49
3,26/0.59
3.32/0.67
3.30/0.75
16.39/0.41
3.78/0.51
3.10/0.61
3.06/0.71
3.09/0.82
3.17/0.92
D.2 Additional Quantitative and Qualitative results
In Table we include all the hyperparameters used in our main experiments (Section 4 In Table 7 we report more quantitative results on CIFAR 10 dataset (21). By using "100K", we mean total of 100K sampling trajectories are generated by the teacher model, which equals around 781 training iterations with a batch size of 128. The experiment settings here are basically in accordance with T For each row of the results of SFD-v, the number of reported results corresponds to the the length of the list of sampling steps L in Algorithm 6 For example, in the third row, the SFD-v is trained to sample with 2, 3, 4 and 5 NFE (each NFE is trained for 200K sampling trajectories average), while in the fourth row average). In the last row, we also include results of conditional image generation CIFAR10 with pre-trained model provided by EDM [6
trained to sample with to NEE (100 for each NFE on
Following the final setting in Section 3.2 sampling steps K in Table g fine-tuning time.
we provide an ablation study on the intermediate teacher It is shown that K achieves good trade-off between FID and
18

---

During our experiments, we adhere to the time schedules utilized in previous studies (for instance, a polynomial schedule with p 7 for pixel-space pre-trained models from EDM [6 and linear schedule for latend-space models from LDM [41), and find them effective. In Figure 9 ablation study on CIFAR1O 10 dataset with 2-NFE SFD trained with different polynomial coefficients.
we show an
Table 9: Ablation study time schedule CIFARIO dataset.
Time schedule
FID
$
[80.00, 15.11, 1.22, 0.006)
5,50
6
[80.00, 12.63, 0.86, 0.006]
4.61
7
[80.00, 10.93, 0.67, 0.006]
4,53
8
[80.00, 9.72, 0.55, 0.006]
4.54
180.00, 8.82. 0.47, 0.006]
10 [80.00, 8.13. 0.42, 0.006]
4.81
For further evaluation on fidelity and diversity, we compute precision, recall, density and coverage following standard practice (35] on CIFARI0 dataset. We use the same random seed for a fair comparison. The results are shown Table I0 In general. while achieving considerable acceleration on image generation, method does not sacrifice diversity.
Table 10: Evaluation on fidelity and diversity on CIFARIO 10 dataset.
Method
NFE
FID
Precision
Recall
Density
Coverage
SFD-y (ours)
4.28
0.77
0.70
1.06
0.93
0.78
0.71
0.94
3.18
0.79
0.71
1.13
0.94
5
2.95
0.79
0.71
1.15
0.95
DPM++(3) 30
11
3.93
0.76
0.71
1.04
0.94
15
2.64
0.76
0.73
0.95
19
2.54
0.77
0.72
0.96
23
2.65
0.77
0.72
1.05
0.96
50
2.01
0.78
0.72
1.11
0.96
DDIM 48
2.91
0.79
0.71
1.09
0.95
16
0.79
0.72
0.96
We include more qualitative results from Figure 11 to Figure 18
19

---

Roaher trajectory
laacher Uapodoly
Teacher trgectery
Student tralector
Student snolectory
Student tralecory
1000
3000 4000
10
PC1
1000
3000 Xr,
6000
PC1
1000
Mr,
3000 4000
PC1
snjeectory
PC2
PC2
FC2
2000
4000
15
PC1
2000
3000 R2
4000
15
2000 X4
3000
4000
15
PC1
Teacher trajectory
Teacher trapedorr
Teacher trejctory
Studint trajectory
Student sejectory
Studonk trajectory
1000
K,
2000
3000
10
1000
3000
10
1000
3000
10
Student
Student tonlectory
Student
PC2
PC2
2000
4000
15
PC1
1000
2000
3000
4000
FC1
1000 2000 -X,
4000
15
PC1
Teacher trajectory
Teacher trajectory
Teacher trajectory trapectory
Studant trapectory
Student srajectory
PC2
1000 2000
3000 4000
10
1000
2000
3000
4000
1000 2000 3000
10
Wocher trajectory
Noacher uaadorry
Nocherr trqectory
Studont trajectory
Soudent srjecttrry
Studont trojectory
5
1000 2000
J000
4000
10
PC1
1000
3000
6000
1000
3000 4000
PC1
Figure 11: Visualization of the effectiveness of SFD.
20

---

DDIM 4 steps
DDIM 8 steps
SFD-v (ours) 4 steps
DDIM 4 steps
DDIM 8 steps
SFD-y (ours) 4 steps
Figure 12: Qualitative results generated by Stable Diffusion v1.5 41
21

---

Figure 13: Uncurated qualitative results on CIFARIO. NFE=1.
Figure 14: Uncurated qualitative results on CIFARIO, NFE=3.
22

---

Figure 15: Uncurated qualitative results on ImageNet. NFE=1.
Figure 16: Uncurated qualitative results on ImageNet. NFE=3,
23

---

Figure 17: Uncurated qualitative results on LSUN-Bedroom. NFE=1.
Figure 18: Uncurated qualitative results on LSUN-Bedroom. NFE=3.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions are aligned with the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include all the information needed in Section 4.1 and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: The code with sufficient instructions attached in the supplemental materials and will be open sourced in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nips.cc/public/giee guwde//ddee/// Cote//oodeeeeeseeeeeeee wwwpibmmsssinoooiioooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details of training and test are fully disclosed in Section 4.1 and Table
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We do not report error bars following standard conventions in the related literature. The standard deviation of the FID is small since it is averaged over 50K samples and all the evaluations in our paper share the same random seed, following [66].
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
27

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As mentioned in the main text, we conduct all the experiments using up to 4 NVIDIA A100 GPUs. The required training time fully specified Table and
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't It make it into the paper).
paper (e.g., preliminary or failed experiments that httprppp....cpoolocc///cccccccdd
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurIPS Code Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We didn't describe safeguards in our paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly cite all the used datasets. Our code is based on the repository open sourced by (63] which is licensed according to the Apache License 2.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We include the code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "TrCBDM3p89", "reviewer": "Reviewer_bYyR", "summary": "In this work authors propose a novel diffusion distillation method by unrolling the student model to match unrolled/generated pre-trained diffusion (teacher) model's trajectory. Authors demonstrate effectiveness and compute efficiency of approach on stable diffusion.", "questions": "Though 'N-model' to finetune has been used, it is interesting to see fixing model at some part improves model over all other parts. \nThis could indicate lower curvature part of trajectory and its more a directional feedback which SFD might be exploiting? Can authors further comment on this?\n\nW.r.t trajectory distillation techniques,  Consistency Distillation  can take potentially shortcuts. But PG  retains gradient field. So not sure if such characterization is correct.", "limitations": "As discussed in weakness and questions generalization to other models is unclear, effect of forward process on SFD? And currently this work also lacks conclusive reasoning under what settings its useful also do not demonstrate impact on diversity, etc.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Paper is easy to read and follow.\nShow effectiveness on stable diffusion and much faster to train for large scale stable Diffusion too.\nIt is interesting to see fixing model at some part improves model over all other parts.", "weaknesses": "What is effect of SFD on diversity w.r.t distilled model? It might be easy to have high quality but much lower diversity. Proposed unrolling of student model ( global trajectory optimization) is effectively multi-step training like in structured prediction and imitation learning which has shown to be prone to mode-collapse.\n\nJustification of method is still unclear and also not sure on how sensitive is SFD to training time noise schedule/time step weighting i.e., forward diffusion process and its impact on induced trajectory. E.g. if we assume flow matching style linear trajectories, they could be more easier to distill within proposed framework.\n\nUnrolling trajectories for distillation is also considered in previous works to account for accumulation of error like BOOT, ImagineFlash, etc. So that in isolation can't be considered as major contribution.\n\nAlso proposed method though is more effective for Stable Diffusion, there is siginificant performance drop on LSUN, ImageNet etc w.r.t progressive distillation or consistency distillation? This also further raises questions and need for additional experiments to understand under what settings this method is effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful feedback. Below we address the specific questions.\n\n**Q: What is the effect of SFD on diversity?**\n\nWe appreciate the reviewer’s suggestion to evaluate the diversity of SFD. Following standard practice, we computed fidelity (measured by precision and density) and diversity (measured by recall and coverage) [1] on the CIFAR-10 dataset. The same random seed is used for different solvers in the following for a fair comparison.\n\n|Solver|NFE|FID|Precision|Recall|Density|Coverage|\n|-|-|-|-|-|-|-|\n|SFD-v|2|4.28|0.77|0.70|1.06|0.93|\n||3|3.50|0.78|0.71|1.10|0.94|\n||4|3.18|0.79|0.71|1.13|0.94|\n||5|2.95|0.79|0.71|1.15|0.95|\n|DPM-Solver++(3M)|11|3.93|0.76|0.71|1.04|0.94|\n||15|2.64|0.76|0.73|1.03|0.95|\n||19|2.54|0.77|0.72|1.04|0.96|\n||23|2.65|0.77|0.72|1.05|0.96|\n||50|2.01|0.78|0.72|1.11|0.96|\n|DDIM|50|2.91|0.79|0.71|1.09|0.95|\n|Heun|50|1.96|0.79|0.72|1.10|0.96|\n\nIt is shown that the diversity of SFD (NFE = 2,3,4,5) is shown to be closely matched to that of teachers (DPM-Solver++(3M), NFE = 11,15,19,23) and the mode collapse issue is not observed. Moreover, SFD can even surpass the teacher in terms of fidelity. \n\nWe will add these results in the revised version of this paper.\n\n**Q: Justification of the method and the sensitivity to time schedule.**\n\nOur method shares its rationale with several recent approaches like progressive distillation (PD) [2]. Both PD and our SFD aim to establish shortcuts between various timestamps along the sampling trajectory. The primary advantage of our method, in terms of accelerated distillation, compared to PD, is derived from *focusing optimization efforts solely on the timestamps that are actually used for sampling*. As a result, choosing a time schedule is critical to enhance the robustness of SFD.\n\nDuring our experiments, we adhere to the time schedules utilized in previous studies (for instance, a polynomial schedule with $\\rho=7$ for EDM [3] and a linear schedule for LDM [4]), and find their methods effective. In the following, we show an ablation study based on CIFAR-10 with 2-NFE SFD trained with different polynomial coefficients.\n\n|$\\rho$|Time schedule|FID|\n|-|-|-|\n|5|[80.00, 15.11, 1.22, 0.006]|5.50|\n|6|[80.00, 12.63, 0.86, 0.006]|4.61|\n|7|[80.00, 10.93, 0.67, 0.006]|4.53|\n|8|[80.00, 9.72, 0.55, 0.006]|4.54|\n|9|[80.00, 8.82, 0.47, 0.006]|4.60|\n|10|[80.00, 8.13, 0.42, 0.006]|4.81|\n\n**Q: If we assume flow matching style linear trajectories, they could be more easier to distill within the proposed framework.**\n\nFlow matching [5] does not generate a linear sampling trajectory. Although the forward process of the OT-based flow matching is linear, the backward sampling process is not. Our SFD may benefit from smaller curvature. However, as far as we know, there is currently no direct evidence suggesting the OT-based flow matching indeed reduces the curvature. We will explore this direction in future work.\n\n**Q: Unrolling trajectories for distillation is also considered in previous works to account for accumulation of error like BOOT, Imagine Flash, etc.**\n\nWe respectfully disagree with this. \n\nFirst, BOOT [6] does not unroll the trajectory. It is designed to be data-free but not to address error accumulation. As mentioned in the original paper, ``(Error accumulation) becomes more pronounced in our case due to the possibility of out-of-distribution inputs $\\hat{x}_t$ for the teacher model...''. To mitigate this, the authors propose to uniformly sample the timestamp, and use a high-order Heun's method.\n\nSecond, although the backward distillation proposed in Imagine Flash [7] does unroll the trajectory, our method differs from backward distillation, where only the final prediction of the student model is supervised by the teacher instead of the whole trajectory. Moreover, Imagine Flash (released on May 8, 2024) is better to be considered as a concurrent work to ours (NeurIPS abstract submission deadline: May 15, 2024).\n\n**Q: Performance drop on LSUN and ImageNet w.r.t PD or CD.**\n \nOur goal is to achieve image quality similar to that of progressive distillation (PD) with a largely reduced training time. Given around 100-200x faster training time compared to PD, the performance drop is not significant, e.g., 4.28 (SFD-v) compared to 4.51 (+0.23, PD) on CIFAR-10, 9.47 (SFD-v) compared to 8.95 (-0.52, PD) on ImageNet, and 9.25 (SFD-v) compared to 8.47 (-0.78, PD) on LSUN. Moreover, our SFD (second-stage) consistently outperforms both PD and Guided PD for one-step sampling to a large margin.\n\nAdmittedly, SFD's performance is worse than that of consistency distillation (CD). However, CD requires around 1,000x larger training time, and the performance drop can be compensated by 2-3 more NFEs of SFD. \n\n**Q: Further comments on fixing model at some part improves model over all other parts.**\n\nWe attribute this property to the regularity of the diffusion model trajectory revealed in a recent work [8]. It is shown that the sampling trajectories generated by diffusion models tend to share a simple boomerang shape (Fig. 4 in [8]). The effect of our method is to modify the gradient (originally pointing towards the tangent direction) along the trajectory to build ''shortcuts''. Seeing the model as a nearly continuous function, it is intuitive that when the gradient at time $t$ is modified in a certain direction, the gradient fields at other timestamps could be influenced in a similar trend, since the model parameter is shared. Given the simple boomerang shape, we hypothesize that the gradient fields at most timestamps are modified closer to the directions of the desired ''shortcuts''.\n\nThis observation is primarily shown by our simple experiment in Section 3.1. The extrapolation ability shown in Figure 6 also provides evidence that the performance of SFD outperforms DDIM to a large degree, even with untrained NFE. Moreover, our main results further verify this, showing that SFD-v consistently outperforms SFD under the same averaged training iterations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wDVPiUcwsh", "reviewer": "Reviewer_RDCV", "summary": "This paper proposes a fast distillation method for diffusion models. This method simplifies the existing knowledge distillation framework and proposes Simple and Fast Distillation from a global perspective to reduce redundant time steps in training. The SFD framework can achieve good experimental results in a very short time compared to existing methods.", "questions": "1.\tAdd more explanations to some meaningful figs in the paper.\n2.\tConduct more ablation experiments or explain the criteria for selecting some hyperparameters in the method.\n3.\tAdding more experiment as I mentioned in weakness 4.", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe method proposed in the paper is not complicated and explores many available improvement ideas, achieving good results.\n2.\tCompared with existing methods, the method proposed in this paper can achieve good results in an extremely short time, with significant improvements in time efficiency.", "weaknesses": "1.\tMany of the techniques in the paper lack innovation and are more like technical explorations.\n2.\tSome of the figures in the paper are difficult to understand, and figures like fig.6 should be explained more.\n3.\tThe selection of some hyperparameters in the paper, such as t_min, appears too random.\n4.\tIf the performance of other training methods such as CTM under the same training time as SFD were reported, it will further demonstrate the superiority of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback! Below we address the specific questions.\n\n**Q: Many of the techniques in the paper lack innovation and are more like technical explorations.**\n\nHere we would like to clarify the main technical contributions of this paper.\n\n(1) In Section 3.1, for the first time, we recognize that distillation-based training makes a smooth modification of the gradient field of diffusion models. Motivated by this, we propose to fine-tune on a coarse-grained timestamps to reduce the large training overhead typically required for distillation-based methods while achieving decent performance. \n\n(2) In Section 3.2, we view distillation from a global perspective (SFD), which is well-motivated by the defects of local fine-tuning as analyzed in the main text. \nTo release the potential of our method, several technical explorations are investigated, e.g., efficient solvers, timestamps and loss metrics. As verified by our main experiments, the obtained settings (displayed in Table 6) are robust across different datasets.\n\n(3) The Section 3.3 involves another technical contribution where we propose improved SFD-v with the step-condition, and it consistently outperforms SFD under the same averaged training iterations, providing further evidence to our innovative finding in Section 3.1. \n\n(4) In Section 3.4, we offer a novel strategy for text-to-image distillation motivated by the shape of latent trajectories, which is also shown to be effective.\n\n**Q: Explanation on Figure 6.**\n\nThe intention of Figure 6 is to provide a further evidence to the smooth modification mentioned in Section 3.1. Here is the explanation. \n\nTake ``SFD, NFE4'' as example, where the SFD is only trained to sample with NFE of 4. Its performance is marked by a star. Interestingly, when using this SFD to sample with untrained NFEs (i.e., 2,3,5,6), even though the timestamps have never been trained in these cases, the performance is still decent and outperforms DDIM to a large degree (DDIM with NFE of 6 gives a FID of 35.62). \n\nFor ``SFD-v, NFE2-5'', the SFD-v is trained to sample with NFE of 2, 3, 4 and 5 by a single model. The performance is marked by 4 stars. Similarly, good results are observed for untrained NFEs (DDIM with NFE of 10 gives a FID of 15.69). \n\nThis extrapolation ability further verifies our finding that there do exist a mutual enhancement when modifying the gradient fields for different timestamps. Thank you for noting this! We will make it clearer in the revised version. Please let us know if there are any possible ambiguities in this paper.\n\n**Q: The criteria for selecting hyperparameters.**\n\nThe final selected hyperparameters are displayed in Table 6. Here we illustrate the criteria for the first 4 items (i.e., teacher solver, $K$, $t_{\\min}$ and AFS).\nAs validated in Section 3.2, they are set the same for the first 3 datasets. Therefore, we focus on illustrating why they are different for Stable Diffusion. \n\nAs for the teacher solver, it is well-known that for Stable Diffusion, a higher-order solver may suffer from instability especially when the guidance scale is large. Therefore, in the official implementation, DPM-Solver++(2M) is the default setting for DPM-Solvers (mentioned in line 285). We simply follow this setting.\n\n$K$ denotes the number of teacher sampling steps taken from $t_{n+1}$ to $t_n$, and we provide an ablation study of $K$ for CIFAR-10 in Table 8. We find that $K=4$ achieves a good trade-off between performance and fine-tuning time and thus adopt this for other two datasets. As Stable Diffusion requires a significantly larger resources, we re-evaluate this setting and choose $K=3$ in this case.\n\nWe use $t_{\\min} = 0.1$ for Stable Diffusion because of its different default time schedule. While $t_{\\min} = 0.002$ is used in EDM by default, $t_{\\min} \\approx 0.03$ is used in Stable Diffusion. Following the same criteria shown in Figure 5, we find $t_{\\min} = 0.1$ to be a robust setting for the teacher solver to give consistently better results than the default setting.\n\nFinally, it is empirically observed that the approximation error made by AFS is not negligible for Stable Diffusion when the guidance scale is large. This is intuitive given the complex trajectories shown in Figure 9c. Since large guidance scale is typically used in practice, we disable AFS for Stable Diffusion.\n\n**Q: The performance of other training methods such as CTM under the same training time as SFD.**\n\nAs mentioned in Appendix B.1, we follow the setting used for CTM [1] faithfully to estimate the training time. \nSince the pre-trained CIFAR-10 models used for both SFD and CTM are from EDM [2], a direct comparison can be made given the FID-iteration curve shown in [1] (Figure 15 of [1]).\n\nOur SFD-v shown in Table 2 is trained for 4.26 A100 hours, which is equivalent to around 7,000 CTM training iterations. \nAccording to the referred FID-Iteration curve, a FID of around 3.4 is achieved by CTM with 18 NFEs while 3.18 is obtained by SFD-v with only 4 NFEs. \n\nOur SFD (second-stage) is trained for 4.88 A100 hours, which is equivalent to around 8000 CTM iterations. \nA FID of around 7.5 is achieved by one-step CTM while 5.83 is obtained by ours. \nTo reach a FID around 6, 40,000 iterations are required for CTM without GAN loss, which costs around 24 A100 hours.\n\n**Reference:**\n\n[1] Kim D, Lai C H, Liao W H, et al. Consistency trajectory models: Learning probability flow ode trajectory of diffusion[J]. arXiv preprint arXiv:2310.02279, 2023.\n\n[2] Karras T, Aittala M, Aila T, et al. Elucidating the design space of diffusion-based generative models[J]. Advances in neural information processing systems, 2022, 35: 26565-26577.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TevRnbpEUd", "reviewer": "Reviewer_4y5W", "summary": "This paper presents a fast-training accelerate sampling algorithm that is based on the distillation paradigm. This algorithm performs trajectory matching under all time points from 1 to 0 (t=80-0.006). The approach avoids the huge overhead of bi-level optimization through the ``detach()'' operation in pytorch, and further improves the performance of the algorithm by correcting the settings of a series of hyperparameters: loss function ($L_1$), step-condition, analytical first step.", "questions": "1. Can SFD accelerate DiT? It would be beneficial to include some preliminary experiments to explore this possibility.\n\n2. Many accelerated sampling algorithms for diffusion models employ discriminators for trajectory matching, with LCM being a prominent example. Could this discriminator-based approach outperform $L_1$ loss in the context of SFD?\n\n3. Is there a bound for SFD's performance as training time is extended? Given that matching in the form of global inherently involves compression by discarding randomness.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The results of the algorithm presented in this paper are extremely impressive, surpassing all known distillation-based accelerated sampling algorithms.\n\n2. The experiments conducted in this paper are thorough and well-analyzed. Particularly enlightening is the analysis of the guidance scale on Stable Diffusion.\n\n3. The paper is exceptionally well-written, offering clarity and ease of understanding, and it is inspiring in various ways. Notable examples include the introduction of the SFD algorithm and the innovative use of CFG=1 for training, as well as the adaptation of any CFG for inference on SD-v1.5.", "weaknesses": "The paper has no significant shortcomings; however, it is worth noting that while the algorithm substantially reduces training costs, it also inevitably lowers the maximum performance threshold. An open question remains: if we extend the training duration, can we achieve results comparable to those of the CTM? It appears that not all time steps contribute equally to model performance. Additionally, introducing randomness through local trajectory matching could potentially enhance generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback! Below we address the specific questions.\n\n**Q: If we extend the training duration, can we achieve results comparable to those of the CTM?**\n\nThe remarkable FID results of CTM are mainly attributed to the introduced GAN loss, otherwise the FID would significantly increase.\nIn Table 2, the results of our SFD (second stage) are on par with CTM without the GAN loss. However, as we will elucidate later, the goal of discriminator-based approaches is generally in contradiction with the goal of trajectory distillation methods such as progressive distillation and our SFD.\n\n**Q: Could discriminator-based approach outperform L1 loss?**\n\nIf we do not get it wrong, the mentioned discriminator-based approach refers to the GAN loss used in CTM [1], because we do not see a discriminator used in the LCM paper [2]. Generally, the effect of this GAN loss is to match the quality of the denoised sample to a real image, which can be treated as building a ''shortcut'' from $x_t$ to $x_0$ for every sampled $t$. However, this does not align well with the goal of trajectory distillation, where we are to build a ''shortcut'' from $x_t$ to $x_s$ ($t > s$) for sampled $t$ and $s$, aiming at faithfully matching the teacher trajectory. Only when $s=0$ will these two objectives align. Therefore, trajectory distillation methods could hardly benefit from this GAN loss.\n\n**Q: Can SFD accelerate DiT?**\n\nSurely. One appealing property of diffusion models is the unique encoding, or referred to reproducibility in [3]. Given a noise distribution and a dataset, diffusion models build a fixed mapping between the noise and the implicit data distribution, regardless of different model architectures (e.g., DiT, U-ViT or U-Net) and training procedures, as long as the model capacity and data size are promised [3]. Starting from the same noise, the sampling trajectory given by a U-Net could resemble that of a DiT, as both of them are trained to predict the same score function. Therefore, the effectiveness of SFD should be independent to the model architecture.\n\nWe conduct some primary experiments using the provided DiT-XL/2 model on ImageNet 256x256. We train our SFD with settings of DPM-Solver++(2M) teacher, $K=2$, 100K generated trajectories.\nSince DiT adopts classifier-free guidance, AFS is disabled. \nWe use guidance scale of 4 and 10,000 images for FID evaluation. The results are shown below.\n\n|Solver|NFE|FID-10K|\n|-|-|-|\n|SFD|3|14.34|\n|DPM-Solver++(2M)|9|13.58|\n|DDIM|3|56.64|\n\n**Q: Introducing randomness through local trajectory matching could potentially enhance generalization. Is there a bound for SFD's performance as training time is extended?**\n\nTo show that SFD (global trajectory matching) does not sacrifice generalization, we compute fidelity (measured by precision and density) and diversity (measured by recall and coverage) [4] on CIFAR-10 dataset following the standard practice. Below are the results.\n\n|Solver|NFE|FID|Precision|Recall|Density|Coverage|\n|-|-|-|-|-|-|-|\n|SFD-v|2|4.28|0.77|0.70|1.06|0.93|\n||3|3.50|0.78|0.71|1.10|0.94|\n||4|3.18|0.79|0.71|1.13|0.94|\n||5|2.95|0.79|0.71|1.15|0.95|\n|DPM-Solver++(3M)|11|3.93|0.76|0.71|1.04|0.94|\n||15|2.64|0.76|0.73|1.03|0.95|\n||19|2.54|0.77|0.72|1.04|0.96|\n||23|2.65|0.77|0.72|1.05|0.96|\n||50|2.01|0.78|0.72|1.11|0.96|\n|DDIM|50|2.91|0.79|0.71|1.09|0.95|\n|Heun|50|1.96|0.79|0.72|1.10|0.96|\n\nIt is shown that the diversity of SFD is very close to that of the teachers, meaning that SFD generalizes well.\nAs for the performance bound, we extend the training iterations of SFD-v shown in Table 2 up to four times. Below are the FID results for different trained trajectories and NFEs. \n\n|Trained trajectories|NFE=2|NFE=3|NFE=4|NFE=5|\n|-|-|-|-|-|\n|800K|4.28|3.50|3.18|2.95|\n|1600K|4.28|3.47|3.11|2.90|\n|2400K|4.18|3.41|3.05|2.92|\n|3200K|4.24|3.40|3.04|2.91|\n\n**Reference:**\n\n[1] Kim D, Lai C H, Liao W H, et al. Consistency trajectory models: Learning probability flow ode trajectory of diffusion[J]. arXiv preprint arXiv:2310.02279, 2023.\n\n[2] Luo S, Tan Y, Huang L, et al. Latent consistency models: Synthesizing high-resolution images with few-step inference[J]. arXiv preprint arXiv:2310.04378, 2023.\n\n[3] Zhang H, Zhou J, Lu Y, et al. The emergence of reproducibility and consistency in diffusion models[C]//Forty-first International Conference on Machine Learning. 2023.\n\n[4] Naeem M F, Oh S J, Uh Y, et al. Reliable fidelity and diversity metrics for generative models[C]//International Conference on Machine Learning. PMLR, 2020: 7176-7185.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a new diffusion distillation approach that significantly reduces training and fine-tuning times by letting a student learn a full teacher trajectory (rather than learning to mimic a single step), reducing the overall distillation error. 

The reviewers appreciated the strong empirical performance (reduced training costs) of the algorithm and the careful analysis carried out in the experiments part. Among weaknesses, reviewers pointed out that the paper could have done a better job at discussing the tradeoffs between runtime and sample diversity as well as the performance limits of the distillation approach. This concern was successfully resolved in the discussion phase, where the authors presented additional evidence supporting their claims. The reviewers recommended acceptance at the end of the discussion phase, and the AC agreed with this outcome. 

Initial criticisms included the tradeoffs of faster generation on sample diversity. However, the authors successfully addressed this concern in the rebuttal period. As an aside, the authors should incorporate discussions on more recent work on solver-based methods (without running experiments) [1,2].

[1] Gonzalez, M., Fernandez Pinto, N., Tran, T., Hajri, H., & Masmoudi, N. Seeds: Exponential SDE solvers for fast high-quality sampling from diffusion models. Advances in Neural Information Processing Systems, 2023.

[2] Pandey, K., Rudolph, M., & Mandt, S. Efficient Integrators for Diffusion Generative Models. In International Conference on Learning Representations, 2024.
Decision: Accept (poster)