Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation
Authors: Shengyuan Zhang
Ling Yang
Zejian Li
An Zhao
Chenye Meng
Changyuan Yang
Guang Yang
Zhiyuan Yang
Lingyun Sun
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Model
Diffusion Distillation
One-step Generation
Primary area: generative models
TL;DR: This paper proposes an efficient and fast distillation method for diffusion models by introducing the convergence trajectory.

Abstract: Accelerating the sampling speed of diffusion models remains a significant challenge. Recent score distillation methods distill a heavy teacher model into a student generator to achieve one-step generation, which is optimized by calculating the difference between two score functions on the samples generated by the student model.
However, there is a score mismatch issue in the early stage of the score distillation process, since existing methods mainly focus on using the endpoint of pre-trained diffusion models as teacher models, overlooking the importance of the convergence trajectory between the student generator and the teacher model.
To address this issue, we extend the score distillation process by introducing the entire convergence trajectory of the teacher model and propose $\textbf{Dis}$tribution $\textbf{Back}$tracking Distillation ($\textbf{DisBack}$). DisBask is composed of two stages: $\textit{Degradation Recording}$ and $\textit{Distribution Backtracking}$. 
$\textit{Degradation Recording}$ is designed to obtain the convergence trajectory by recording the degradation path from the pre-trained teacher model to the untrained student generator.
The degradation path implicitly represents the intermediate distributions between the teacher and the student, and its reverse can be viewed as the convergence trajectory from the student generator to the teacher model.
Then $\textit{Distribution Backtracking}$ trains the student generator to backtrack the intermediate distributions along the path to approximate the convergence trajectory of the teacher model.
Extensive experiments show that DisBack achieves faster and better convergence than the existing distillation method and achieves comparable or better generation performance, with an FID score of 1.38 on the ImageNet 64$\times$64 dataset.
DisBack is easy to implement and can be generalized to existing distillation methods to boost performance.

Published as a conference paper at ICLR 2025
DISTRIBUTION BACKTRACKING BUILDS A FASTER CONVERGENCE TRAJECTORY FOR DIFFUSION DISTIL- LATION
Shengyuan Zhang", Ling Yang', Zejian Li An Zhao', Chenye Meng', Changyuan Yang', Guang Yang'. Zhiyuan Yang'. Lingyun Sun
Coollege of Computer Science and Technology. Zhejiang University
Schooo of Software Technology, Zhejiang University
*Peking University
Allibaba Group
1,2 (zhangshengyuan, zejianlee, zhaoan mengcy, sunly)ezju, edu. en
[yangling0818)@113 com
{ "changyuan, yangcy, qingyun, adam. admam adam.yay ien.iinc.ooo com
Corresponding author
ABSTRACT
Accelerating the sampling speed of diffusion models remains significant chal- lenge. Recent score distillation methods distill a heavy teacher model into a student generator to achieve one-step generation, which is optimized by calcu- lating the difference between two score functions on the samples generated by the student model. However, there is score mismatch issue in the early stage of the score distillation process, since existing methods mainly focus on using the endpoint of pre-trained diffusion models teacher models, overlooking the importance of the convergence trajectory between the student generator and the teacher model. address this issue, we extend the score distillation process by introducing the entire convergence trajectory of the teacher model and pro- pose Distribution Backtracking Distillation (DisBack). DisBask is composed of two stages: Degradation Recording and Distribution Backtracking. Degra- dation Recording is designed to obtain the convergence trajectory by recording the degradation path from the pre-trained teacher model the untrained student generator. The degradation path implicitly represents the intermediate distribu- tions between the teacher and the student, and its reverse can be viewed the convergence trajectory from the student generator to the teacher model. Then Distribution Backtracking trains the student generator to backtrack the interme- diate distributions along the path to approximate the convergence trajectory of the teacher model. Extensive experiments show that DisBack achieves faster and better convergence than the existing distillation method and achieves compara- ble or better generation performance, with an FID score 1.38 on the ImageNet 64x64 dataset. DisBack is easy implement and be generalized to exist- ing distillation methods to boost performance. Our code publicly available on http://wwww...cccc//oooo/////aaa
INTRODUCTION
Recently, generative models have demonstrated remarkable performance across diverse domains such as images (Kou et al., 2023 Yin et al.] (024a)) audio (Evans et al.) 2024 Xing et al.) (2024), and videos (Wang et al. 2024 Chen et al.) 2024). However, existing models still grapple with the "trilemma" problem, wherein they struggle to simultaneously achieve high generation quality, fast generation speed, and high sample diversity (Xiao et al., (021). Generative Adversarial Networks (GANs) (Goodfellow al. 2014) rapidly produce high-quality samples but often face mode collapse issues. Variational Autoencoders (VAEs) (Kingma & Welling 2014) offer stable training but tend to yield lower-quality samples. Recently, Diffusion models (DMs) have emerged

---

Published as a conference paper at ICLR 2025
200
DisBack Diff-Instruct
200
DisBack
160
180
DisBack
175
175
Iift-lnstruct
140
Iiff-un
150
150
120
125
125
X
100
100
2 80
75
75
60
50
x
50
1.97
1.79x
25
20
200
1000
2000
0.0
1000
2000
0,000
1000
2000
Epochs
Epochs
Epochs
(a)
(b)
(c)
Figure 1: The comparison of the distillation process between existing SOTA score distillation method Diff-Instruce (Luo al. 2023c) and proposed DisBack on (a) CIFARIO, (b) FFHQ 64x64, and (c) ImageNet 64x64 datasets. The first 200 epochs refer to the computational overhead of the degradation recording stage of DisBack. DisBack achieves a faster convergence speed due to the constraint of the entire convergence trajectory between the student generator and the teacher model.
competitive contender in the generative model landscape (Fan et al., 2023 Zhou et al. 2023 Xu
jet al. 2024). Diffusion models can generate high-quality, diverse samples but still suffer from slow sampling speeds due to iterative network evaluations.
To accelerate the sampling speed, the score distillation method tries to distill heavy teacher model to student generator to reduce the sampling cost and achieve the one-step generation (Bao et al.)
2023 Luo et al. 2023c; Yin et al.) (024b). The score distillation method optimizes the student generator by calculating the difference between two score functions
the samples generated by the student generator. However, as the generated distribution is far from the training distribution at the beginning, the generated sample lies outside the training data distribution. Thus, the predicted score of the generated sample from the teacher model does not match the sample's 's real score in the training distribution. This mismatch issue is reflected by unreliable network predictions of the teacher model, which prevents the student model from receiving accurate guidance and leads to a decline in final generative performance. We identified that this issue arises because existing score distillation methods mainly focus on using the endpoint of the pre- pre-trained diffusion model as the teacher model, overlooking the importance of the convergence trajectory between the student gener- ator and the teacher model. Without the constraint of the convergence trajectory, the mismatch issue causes the student generator to deviate from reasonable optimization path during training, leading to convergence to suboptimal solutions and decline in final performance.
To address this problem, we extend the score distillation process by introducing the entire conver- gence trajectory of the teacher model and propose Distribution Backtracking Distillation (DisBack) for a faster and more efficient distillation. The construction of DisBack is based on the following insights. In practice, the convergence trajectory of most teacher models is inaccessible, particularly for large models like Stable Diffusion (Rombach et al. (022). Because the trajectory of distribution changes is bidirectional, it is possible to construct degradation path from the teacher model the initial student generator, and the reverse of this path can be viewed as the convergence trajectory of the teacher model. Compared with fitting the teacher model directly, fitting intermediate targets along the convergence trajectory can mitigate the mismatch issue. Thus, the DisBack incorporates degradation recording and distribution backtracking stages. In the degradation recording stage, the teacher model is tuned to fit the distribution of the initial student generator and obtains a distribu- tion degradation path. The path includes a series of in be-between diffusion models to represent the intermediate distributions of the teacher model implicitly. In the distribution backtracking stage, the degradation path is reversed and viewed as the convergence trajectory. Then the student generator is trained to backtrack the intermediate distributions along the path to optimize towards the conver- gence trajectory of the teacher model. In practice, the degradation recording stage typically requires only a few hundred iterations. Therefore, the proposed method incurs trivial additional computa- tional costs. Compared to the existing score distillation method, DisBack exhibits a significantly increased convergence speed (Fig. [ and it also delivers superior generation performance (Fig. 2
Our main contributions are summarized as follows. (1) We extend the score distillation process by introducing the constraint of the entire convergence trajectory of the teacher model and propose Dis-
2

---

Published as a conference paper at ICLR 2025
gold, glass battic, pirate salling on a leaf.
pastries this white plate.
space
Phoenix emerging from fire with galaxy.
Magical world, valley.
quantic vintage Futurist, Space
A floating island level suspended in
The jaker walking through strects of York.
A Adog laying on its stomach
Figure 2: Several examples of 1024x 1024 images generated by our proposed one-step DisBack model distilled from SDXL [Podell et al. 2024).
tribution Backtracking Distillation (DisBack), which achieves faster and efficient distillation (Sec. 4. (2) Extensive experiments demonstrate that the proposed DisBack accelerates the conver- gence speed of the score distillation process while achieving comparable or better generation quality compared to existing methods (Sec. 5). (3) The contribution of DisBack is orthogonal to those of other distillation methods. Researchers are encouraged to incorporate our DisBack training strategy into their distillation methods.
2 RELATED WORKS
Efficient diffusion models. To improve the efficiency of the diffusion model, existing methods use knowledge distillation to distill a large teacher model to a small and efficient student model Yang et al. 2022). The progressive distillation (Salimans Ho 202I progressively distills the en- tire sampling process into a new diffusion model with half the number of steps iteratively. Building on this, the classifier-guides distillation (Sun al. (023) introduces a dataset-independent classifier to focus the student on the crucial features to enhance the distillation process. Guided-distillation (Meng et al. 2023) proposes a classifier-fres guiding framework to avoid the computational cost of additional classifiers and achieve high-quality sampling in only 2-4 steps. Recently, the Consistency Model (Song et al. 2023) uses the self-consistency the ODE process to achieve one-step distil- lation, but this is at the expense of generation quality. To mitigate the surface of the sample quality caused by the acceleration, the Consistency Trajectory Model (Kim et al.) (024a) combines the ad- versarial training and denoising score matching loss to further improve the performance. Latent

---

Published as a conference paper at ICLR 2025
Adversarial Diffusion Distillation (Sauer et al. 2024) leverages generative features from pre-trainnn trained latent diffusion models to achieve high-resolution, multi-aspect ratio, few-step image generation.
Score distillation for one-step generation. Diff-Instruct (Luo et al. 2023c) distills pre-trained diffusion model into a one-step generator by optimizing it with the gradient of the difference be- tween two score functions-one for the pre-trained diffusion distribution and the other for the gen- erated distribution. Adversarial Score Distillation (Wei et al.) 2024) further employs the paradigm of WGAN and retains an optimizable discriminator to improve performance. Additionally, Swift- brush (Nguyen & Tran 2024) leverages score distillation to distill a Stable Diffusion v2.1 into a one-step generator and achieve competitive results. DMD (Yin et al., 2024b) suggests the inclu- sion of a regression loss between noisy images and corresponding outputs to alleviate instability in the distillation process in text-to-image generation tasks. DMD2 (Yin et al., (024a) introduces a two-time-scale update rule and additional GAN loss address the issue of generation quality being limited by the teacher model in DMD, achieving superior performance. Recently, HyperSD (Ren et al. 2024) integrates score distillation with trajectory segmented consistency distillation and human feedback learning, which achieves SOTA performance from to inference steps.
Mismatch issues in diffusion model. Existing studies have already discussed the issue of score mismatch in diffusion models Chao et al. (2022); Kim et al. (2024b). DLSM [hao et al. (2022) deeply analyzed the mismatch between the posterior score estimated by diffusion models and the true likelihood score in conditional generation and proposed the Denoising Likelihood Score Matching loss train classifiers for more accurate conditional score estimation. TIW-DSM|Kim al. (2024b) highlighted that dataset biases cause mismatches between the generator's s and true data distributions, introducing varying importance weights and score correction terms to mitigate this issue by assigning different weights to data points at each time step and correcting the scores. This paper focuses on the mismatch arising from the disparity between the initial generator's 's distribution and the teacher model's 'ss training distribution, causing discrepancies between predicted and true scores.
3 PRELIMINARY
In this part, we briefly introduce the score distillation approach. Let de and gf be the distribution of the student generator Gsta and its noisy distribution at timestep In addition, qo and qe are the training distribution and its noisy distribution at timestep L. By optimizing the KL divergence in Eq. (T). we can train a student generator to enable one-step generation (Wang et al. 2023).
min DEL (90 (xo) ||go (xo))
"
Here Tp = Gatu(2: the generated samples, and 17 is the trainable parameter of Gstu- How- ever, due the complexity of go and its sparsity in high-density regions, directly solving Eq. is challenging (Song & Ermon, 2019). Inspired by Variational Score Distillation (VSD) (Wang et al., 2023]. {I) can be extended to optimization problems at different timesteps in Eq. (2). As increases, the diffusion distribution becomes closer to a Gaussian distribution.
min B...H. DKL (q. (ax)) I|ge (zx)) "
(2)
Here is the noisy data and o Zo NZz,, of Theorem proves that introducing the additional KL-ddiveggence for > 0 does not affect the global optimum of Eq.
Theorem (The global optimum of training (Wang et al. 2023)) Given t >>> 0, have
DKL (g (ax) 41 (a:)) = <> DKL (96 (mr) l|go (2ro)
Therefore, by minimizing the KL divergence in Eq. the student generator can be optimized through the following gradients:
dx; On
v, DKL (g. (xzz) llge (zx:) E,, (V log df (æx) Vz, log (a:))
(4)
Here the score perturbed training data V... log qe (xx) can be approximated by a pre- trainrd dif- fusion model so. The score of perturbed generated data Va, log of (æ.) is estimated by another
4

---

Published as a conference paper at ICLR 2025
Stage Degradation Recording
Pre-trained
Degraded
0x
Distilled
Distilled
Distilled
Fits
N-
Go
edu
atu
Optimal Generator
Initial Generator
Stage Distribution Backtracking
An auxiliary diffusion model initialized with the teacher model $a and then fits the distribution of the initial student generator G The in- termediate checkpoints {s/g
Figure 3: The overall framework of DisBack. Stage I:
} ........ are saved to form degradation path. The degradation path is then reversed and viewed as the convergence trajectory. Stage 2: The intermediate node so, along the convergence trajectory is distilled the student generator sequentially until the generator converges to the distribution of the teacher model.
diffusion model $4, which is optimized by score matching with generated data (Song et al.
2021bb
3D 22 o?
min Er,e 86 (I),
(5)
Thus, the gradient of student generator in (4. estimated
0xz On
DKL (ge (are) |Ige (ar+) Bec [so (xe,() (xte 0)
(6)
The distribution of the student generator changes after update. Therefore, 86 also needs to be optimized based on the newly generated images ensure the timely approximation of the generated distribution. Thus, the student generator and sg are optimized alternately.
In practice, So has three initialization strategies: (1) s. is randomly initialized (Franceschi et al, 2023). (2) 8g is initialized as 88 or its LoRA (Hu et al. 2021 Wei et al. 2024). (3) 8 is initialized
by fitting the generated samples of student generator (Luo al. 2023c). Beyond unconditional
image generation (Ye & Liu (003), this method has also been applied to tasks such as text-to-image
and image-to-image generation across various structures (Yin al. 2024b Hertz al. 2023
METHOD
4.1 INSIGHT
In this section, we introduce the Distribution Backtracking Distillation (DisBack). The key insight behind DisBack is the importance of the convergence trajectory. As mentioned in See there are two score functions in score distillation, one representing the pre-trained diffusion distribution and the other representing the generated distribution. The student model is optimized using the gradi- ent of the difference between these two score functions. Existing methods (Luo et al.) 2023c Yin eet al. 2024b]a) directly use the endpoint of the pre-trained diffusion model as the teacher model, overlooking the intermediate convergence trajectory between the student generator and the teacher model. The resulting score mismatch issue between the predicted scores of the generated sample from the teacher model and the real scores causes the student model to receive inaccurate guidance. ultimately leads to a decline in final performance. Constraining the convergence trajectory be- tween the student generator and the teacher model during the distillation process mitigate the

---

Published as a conference paper at ICLR 2025
mismatch issue and help the student generator approximate the convergence trajectory of teacher models to achieve faster convergence. In practice, is infeasible to obtain the convergence trajec- tory of most teacher models, especially for large models such Stable Diffusion (Rombach et al. 2022). Reversely, it is possible to obtain the degradation path from the teacher model to the initial student generator. The reverse of this degradation path can be viewed as the convergence trajectory of the teacher model. Based on the above insights, we structure the proposed DisBack in two stages including the degradation recording stage and the distribution backtracking stage (Fig. 3).
4.2 DEGRADATION RECORDING
This stage aims to obtain the degradation path from the teacher to the initial student generator. The degradation path is then reversed and viewed as the convergence trajectory of the teacher model. The teacher here is the pre-trained diffusion model se and the student is represented by Gstu.
Let s/ be a diffusion model initialized with the teacher model se, and it is trained on generated samples to fit the initial student generator's distribution of with Eq. (7). By saving the multiple intermediate checkpoints during the training, we can obtain a series of diffusion models (,6,
.,..., JN. where s/n se go and K'y R o These diffusion models describe the scores of non non-existent distributions on the path, recording how the training distribution qu degrades to the initial generated distribution q6- Algorithm I shows the process of obtaining the degradation path. Since distribution degradation is easily achievable, the degradation recording stage only needs trivial additional computational resources (200 iterations in most cases).
30 2t
min I
s's (xt, t)
of
Algorithm Degradation Recording.
Algorithm 2 Distribution Backtracking.
Input:
Initial student generator G stu and pre-trained diffusion model 8g.
Input:
Initial student generator G and re- verse path checkpoints {%6, N N,...,0) 0}
Output:
Degradation path checkpoints O....,, N)
Output: One-step generator G
(%)
S.p <- S for i + N 1 to 0 do
%/8 80
while not converge do
2o (o
while not converge do
Co
G (2;
Update v) with gradient
Er,e [5o (#2, t) s,b, (ar,,) t)] Update e with gradient
Ou On
Update 0 with gradient
E J0.
Fr, % (z)
of 1
80 Br.c (s) (xx, 1)
20-
o7 2
Save intermediate checkpoints 5662 end while
end while end for
4.3 DISTRIBUTION BACKTRACKING
Given the degradation path from the teacher model to the initial student generator, the reverse path is viewed representation of the convergence trajectory between the initial student generator Go and the teacher model 8p- The key to the distribution backtracking is to sequentially distill checkpoints in the convergence trajectory into the student generator. The last node sey, in the path close to the initially generated distribution o6 Therefore, in the distribution backtracking stage, we use s/x_, as the first target to distill the student generator. When near convergence, we switch the target 5/yx 2' The checkpoints s'6 sequentially distilled Gsta until the final target s/ is reached. During the distillation, the gradient of G stu
dx, Grad(1) = E,,, [sS (2re, 1) s66, (wt, 1)]
(8)
On

---

Published as a conference paper at ICLR 2025
Table 1: The unconditional generation performance of DisBack. The FID ()) scores are shown.
Model
NFE (1)
FFHQ
AFHQv2
LSUN-bedroom
LSUN-cat
DDPM (Ho et al. 2020
1000
3.52
4.89
17.10
ADM Dharwwal & Nichol 2021
1000
1.90
5.57
NSCN++ Song et al.
2021b
79
25.95
18.52
+ Song 2022
2021b
79
3.39
2.58
EDM (Karras et al.
79
2.39
1.96
3.57
6.69
EDM (Krrra al.
2022
15
15.81
13.67
Diff Instruct (Luo et al. 2023c
1
19.93
PD (alimmans & Ho 2021
16.92
29.60
CT
Song al. 2023
16.00
20.70
Song et al.
12.58
10.75
7.80
11.00
Dick
10.88
9.97
6.99
10.30
In this stage, Gstu and Sg are also optimized alternately and the optimization of ss is the same as in the original score distillation (Eq.5). Compared to existing score distillation methods, the final target of DisBack the same while constraining the convergence trajectory achieve more efficient distillation of the student generator. Algorithm summarizes the distribution backtracking stage.
5 EXPERIMENT
Experiments are conducted on different models across various datasets. We first compare the per- formance of DisBack with other multi-step diffusion models and distillation methods (Sec. 5.T) Secondly, we compare the convergence speed of DisBack with its variants without the constraint of the convergence trajectory (Sec. 5.2]. Thirdly, further experiments are conducted demonstrate DisBack's 's effectiveness in mitigating the score mismatch issues (Sec. 5.3 Then, also conduct the ablation study to show the effectiveness of introducing the convergence trajectory (Sec. [.4). Finally, we show the results of DisBack on text-to-image generation tasks (Sec.5.5)
5.1 QUANTITATIVE EVALUATION
DisBack can achieve performance comparable or even better than the existing diffusion models or distilla- tion methods. Experiments are con- ducted different datasets. (1) The unconditional generation on FFHQ 64x64, AFHQv2 64x64, LSUN- bedroom 256x256 and LSUN cat 256x256. (2) The conditional genera- tion on ImageNet 64x64. The perfor- mance of DisBack is shown Tab. and Tab. 2 All the DisBack mod- els are distilled from the pre-trained EDM model (Karras et al. 2022).
Table 2: The conditional generation performance of Dis-
Back on ImageNet 64x64 dataset.
Model
NFE (L)
FID (L)
DDPM (Ho et al. 2020
1000
3.77
DDDM (Zhang el al. 2024
1000
2.11
EDM (Krras et al 2022
511
1.36
EDM (Krrras et al. 2022
15
10.46
Moment Matching Sahmans al. 2024
8
3.3
SlimFlow iZhu et al.
2024
12.34
BOOT (Gu et at. 2024
12.30
DDDM (Zhang et 2024
3.47
CTM (Kim
2024a
2,06
(Zhou et al.
2024
1.52
For unconditional generation, the one-step generator distilled by the DisBack achieves comparable perfor- mance across different datasets com- pared to multi-step generation dif- fusion models. Specifically, it out- performs the original EDM model with 15 NFEs (10.88 of DisBack and 15.81 of EDM on FFHQ64). Compared to existing one-step generators and distillation methods, DisBack achieves optimal performance. For conditional generation, the DisBack achieves the best
DMD2 (Yi el al.
2024a
1.51
Diff- Instruct (Luo et al.
2023c
5.57
PD (Salimans & Ho
8.95
CT (Sng 2023
13,00
Song et al.
6.20
DisBack
1.38

---

Published as a conference paper at ICLR 2025
performance compared to the existing models. Moreover, DisBack requires training data and additional constraints during training. In conclusion, DisBack can achieve competitive distillation performance compared to existing models.
5.2 CONVERGENCE SPEED
We conducted a series of experiments to demonstrate the advantages of DisBack in accelerating the convergence speed of the score distillation process on unconditional CIFARIO (Krizhevsky 2009). FFHQ 64x64 (Karras et al. 2019). and conditional ImageNet 64x64 (Dng et al.) 2009) datasets. Diff-Instruct (Luo et al. (023c) the existing SOTA score distillation method, which can be regarded as variation of DisBack not introducing the convergence trajectory. We compared the FID trends DisBack and Diff- during the distillation process in the same situation.
The results are shown in Fig. [ As for unconditional generation, DisBack achieves convergence speed 1.97 times faster than the variant without the constraint convergence trajectory on the FFHQ 64x64 dataset and 5.67 times faster the CIFARIO dataset. For the conditional generation on the ImageNet 64x64 dataset, DisBack is 1.79 times faster than the variant without the constraint of convergence trajectory. The fast convergence speed is because constraining the convergence trajectory of the generator provides a clear optimization direction, avoiding the generator falling into suboptimal solutions and enabling faster convergence the target distribution.
5.3 EXPERIMENTS ON SCORE MISMATCH ISSUE
In this part, experiments are conducted to validate the positive impact of constraining the conver- gence trajectory on mitigating the mismatch issues. We propose a new metric called mismatch degree to assess whether the predicted score of the teacher model matches the distribution's 's real score given data distribution. This score is inspired by the score-matching loss.
dmis
E ma En, (( (0 - Va, log qr (xx
Here 2t is the noisy data from the assessed dis- tribution. Besides, so (are, 1) represents the pre- dicted score and Væ, log q: (x; represents the real score. As shown in Eq. (4) and (6). the closer (ax; 1) is to Vz, log (r:,), the more accurate the gradient estimation for the student generator obtains. Because the real scores are not available in practice, we use Stable Target
0.35
Mismatch degree
0.30
Diff instruct
Ideal lower bound
0.25
0.20
0.15
Field (STF) (Xu et al. 2022) to approximate the real score Vz, log L: (x;) on the assessed distribution. STF estimation leverages refer- ence batches to reduce the variance of training objectives, which has been proven yield ac- curate asymptotically unbiased estimates of the real score (Xu al. (022).
0.10
0.05
0
30000
40000
50000
Epachs
Figure 4: The mismatch degree during the distil- lation process of Diff-Innsrue and proposed Dis- Back. The standard deviation visualized. Dis- Back effectively mitigates the mismatch degree during the entire distillation process.
When the assessed distribution is close to the distribution of the teacher model sg, the mis- match degree is small, and vice versa. When calculated directly on the training data, the re- sulting mismatch degree represents the ideal lower bound. Therefore, the mismatch degree can be used to assess the convergence degree of the generated distribution during the distillation process and visualize the convergence speed under the constraint of the convergence trajectory.
We conduct experiments on the FFHQ 64x64 dataset with Disf-Inntruct (Luo et al. 2023c) as a baseline. We calculate the mismatch degree the distribution of the student generator of both Diff- Instruct and the proposed DisBack. The pre-trained EDM model is chosen as the teacher model. In this scenario, the ideal lower bound of the mismatch degree is 0.037. We visualized the mis- match degree Fig. 4 With the constraining of the convergence trajectory, the mismatch degree of the proposed DisBack is lower during the distillation process, meaning the student generator
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study constraining the convergence trajectory the score distillation process.
The FID (1) scores in each case are shown.
Model
FFHQ
AHFQv2
ImageNet
LSUN-bedroom
LSUN-cat
DisBack
10.88
9.97
1.38
6.99
10.30
w/o Convergence Trajectory
12.26
10.29
5.96
7.43
10.63
A mesmerizing (Unicorn-Horse) craited from swirling fire and lava.
One cute little white puppy sitting an apple orchard autumn
flower, transparent, with Universe within.
Blonde headed eyberpunk girl holding her golden baby dragon.
Portrait photo of a war man cyborg robot in chemical laboratory + face.
Black background with smoke spotight.
stylishly elderly woman face with big glass.
playing water, dog laying shore.
skateboard.
Figure Generation samples by DisBack distilled from SDXL with 1024x 1024 resolution.
verges faster and better. Thus, by constraining the convergence trajectory, the mismatch issue can be mitigated and DisBack can achieve more efficient distillation.
5.4 ABLATION STUDY
Ablation studies are conducted to compare the performance of DisBack with its variant without the constraint of the convergence trajectory. The results are shown in Tab. Results show that the variant without the constraint of convergence trajectory suffers from a performance decay in different cases. This confirms the efficacy of constraining the convergence trajectory between the student generator and the teacher model can improve the final performance of the generation.
5.5 TEXT TO IMAGE GENERATION

---

Published as a conference paper at ICLR 2025
DisBack
LCM
LoRA
castle clif overlooking kingdom.
A glass on dinner table.
some flowers in it.
bedroom with low platform
landscape with snow- covered
Figure 6: One step generation samples by original LCM-LoRA and its variant distilled from SD v1.5 with DisBack 512x512. LCM-LoRA with DisBack can generate images with higher quality.
Further experiments are conducted on text-to-image generation tasks. We use DisBack to distill the SDXL model (Podell et al.) 2024) and evaluate the FID scores the distilled SDXL and the original SDXL on the COCO 2014 (Radford et al., 2021). The user studies are conducted verify the effectiveness of DisBack. We randomly select 128 prompts from the LAION-Aestheties (Schuhmann et al., 2022) to generate images and ask volunteer participants to choose the images they think are better. Detailed information about the user study is included in Sec. B.3 The results of the FID evaluation and user study are presented in Tab. 4 DisBack achieved optimal FID and comparable CLIP scores in single-step generation compared to baselines, but the Patch-FIID showed slight decay. The preference scores of DisBack over the original SDXL. are 61 01.3% Some generation samples are shown in Fig. and Fig. 5
Table 4: The results of text-to-image generation,
Model
NFE (J)
FID (J)
Patch-FID ({)
CLIP (1)
LCM-SDXL
Luo et al.
2023b
81.62
154.40
0.275
LCM-SDXL Luo el
20236
4
22.16
33.92
0.317
DMD2 Yin al 2024a
19.01
26.98
0.336
DMD2
19.32
20.86
0.332
SDXL- Til,-Turoo/an
2025
24.57
23.94
0.337
EDXTT-Turbo)aa /Suter
4125
4
23.19
23.27
0.334
SDXL Lightning Lan a SDXL Lightning Lan at
2024
1
23.92
31.65
0,316
20244
24.46
24.56
0.323
SDXL, Podel 20244
100
21.38
0.332
DisBack
18.96
26.89
0.335
We also conducted experiments on LCM-LoRA (Luo et al. 2023b). The LCM-LoRA distilled from SDv1.5 using DisBack has an FID score of 36.37 on one-step generation, while the FID score of the original LCM-LoRA 78.26. Some generated samples of DisBack and original LCM-LoRA are shown in Fig. 6
The details of experiments and results are provided in Sec. A.1
6 CONCLUSION
Summary. This paper proposes Distribution Backtracking Distillation (DisBack) introduce the entire convergence trajectory of the teacher model in the score distillation. The DisBack can also be used to distill large-scale text-to-image models. DisBack performs faster and more efficient distillation and achieves a comparable or better performance in one-step generation compared to existing multi-step generation diffusion models and one-step diffusion distillation models.
Limitation. The performance of DisBack is inherently limited by the teacher model. The better the original performance of the teacher model, the better the performance of DisBack will also be. Additionally, achieve optimal performances in both accelerated distillation and generation qual- ity, DisBack requires careful design the distribution degradation path and the setting of various hyperparameters (such how many epochs used fit each intermediate node in distribution backtracking stage). While with no meticulous design, can also achieve better performance, fur- ther exploration is required enable the model reach optimal performance.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work is supported by the Provincial Key Research and Development Plan of Zhejiang Province under No. 2024C01250(SDD2), the National Key R&D Program of China under No. 2022YFB330331, and the National Natural Science Foundation of China under No. 62006208. This work is supported by Alibaba-Zhejiang University Joint Research Institute of Frontier Tech- nologies. The authors would like to thank anonymous reviewers for their helpful comments.
REFERENCES
Zhiqiang Bao, Zihao Chen, Changdong Wang. Wei-Shi Zheng, Zhenhua Huang, and Yunwen Chen. Post-distillation via Neural Resuscitation. IEEE Transactions on Multimedia, pp. 3046 3060, 2023.
Chen-Hao Chao, Wei-Fang Sun, Bo-Wun Cheng, Yi-Chen Lo, Chia-Che Chang, Yu-Lun Liu, Yu- Lin Chang, Chia-Ping Chen, and Chun- Lee. Denoising Likelihood Score Matching for Condi- tional Score-based Data Generation. arXiv preprint arXiv: 14206, 2022.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7310-7320, 2024.
Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung-Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8188-81 8197, 2020.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, pp. 8780-8794, 2021.
Zach Evans, CJ Carr, Josiah Taylor, Scott H Hawley, and Jordi Pons. Fast ciming-conditioned latent audio diffusion. arXiv preprint (rxxix:2422.04225 2024.
Wenqi Fan, Chengyi Liu, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang, and Qing Li. Generative diffusion models on graphs: Methods and applications. In International Joint Confer- ence on Artificial Intelligence, pp. 6702-6711, 2023.
Jean-Y Yves Franceschi, Mike Gartrell, Ludovic Dos Santos, Thibaut Issenhuth, Emmanuel de Bézenac, Mickaël Chen, and Alain Rakotomamonjy. Unifying GANs and Score-Based Dif- fusion as Generative Particle Models. In Advances Neural Information Processing Systems, volume 36, pp. 59729-59760, 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.
Jiatao Gu, Chen Wang, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind. Data- free Distillation of Diffusion Models with Bootstrapping. In International Conference on Machine
Learning. 2024.
Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. In International Confer- ence on Computer Vision, pp. 2328-2337, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising Diffusson Probabilistic Models, In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. International Conference on Learning Representations, 2021.

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8110-8119, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Advances in Neural Information Pro- cessing Systems, pp. 852-863, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, pp. 26565- 26577, 2022.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. In International Conference on Learning Representations, 2024a.
Ycongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and II-Chul Moon. Training Unbiased Diffusion Models from Biased Dataset. In The Twelfth Inter- national Conference on Learning Representations, 2024b.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Ziyi Kou, Shichao Pei, Yijun Tian, and Xiangliang Zhang. Character as pixels: a controllable prompt adversarial attacking framework for black-box text guided image generation models. In International Joint Conference on Artificial Intelligence, pp. 4912 2023.
Krizhevsky. Learning multiple layers of features from tiny images. University of Tront, 2009.
Shanchuan Lin, Anran Wang, and Xiao Yang. SDXL-Lightning: Progressive Adversarial Diffusion Distillation. arXiv preprint 2rXiv:2402.13992, 2024.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized GAN training for high-fidelity few-shot image synthesis. In International Conference on Learning Representations, 2020.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04377. 2023a.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint DeXXiv:2311.05555 2023b.
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang, Diff- Instruct: Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models. In Advances in Neural Information Processing Systems, volume 36, pp. 76525-76546, 2023c.
Paul C. Matthews. Vector Calculus. Springer undergraduate mathematics series. Springer, London, 1998. ISBN 978-3-540-76180- doi: doi.org//001007//97. 1-4471-0597-8
Chenlin Meng. Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Thuan Hoang Nguyen and Anh Tran. Swiftbrush: One-step text-to-image diffusion model with variational score distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7807-7816, 2024.
12

---

Published as a conference paper at ICLR 2025
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8888 8171, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high high-resolution image synthesis. In International Conference on Learning Representations, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 7748-8766 PMLR, 2021.
Yuxi Ren, Xin Xia, Yanzuo Lu, Jiacheng Zhang, Jie Wu, Pan Xie, Xing Wang, and Xuefeng Xiao. Hyper-sd: Trajectory segmented consistency model for efficient image synthesis. arXiv preprini 1rX1v:2404.13686 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Tim Salimans and Jonathan Ho. Progressive Distillation for Fast Sampling of Diffusion Models, In International Conference on Learning Representations, 2021.
Tim Salimans, Thomas Mensink, Jonathan Heek, and Emiel Hoogeboom. Multistep Distillation of Diffusion Models via Moment Matching. arXiv preprint arXiv:2406. 240,:24300.,,133,, 2024.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv::403. 12015, 2024.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial Diffusion Distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2025.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Muliis, Mitchell Wortsman, et al. Laion-5b: An open lcage-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 01.25278-25294 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In Inter- national Conference Learning Representations, 2021a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11909 19977, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederík Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In Interna- tional Conference on Learning Representations, 2021b
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency Models. In International Conference on Machine Learning, pp. 3221 -32252,, 2023.
Wujie Sun, Defang Chen, Can Wang, Deshi Ye, Yan Feng. and Chun Chen. Accelerating diffusion sampling with classifier-based feature distillation. International Conference on Multimedia and Expo, pp. 810-815, 2023.
Fei- Yue Wang, Qinghai Miao, Lingxi Li, Qinghua Ni, Xuan Li, Juanjuan Li, Lili Fan, Yonglin Tian, and Qing-Long Han. When does sora show: The beginning of tao to imaginative intelligence and scenarios engineering. JEEE/CAA Journal of Automatica Sinica, 11(4))809 815, 2024.
Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li. Hang Su, and Jun Zhu. Prolific- Dreamer: High-Fidelity and Diverse Text-to-DD Generation with Variational Score Distillation. In Advances Neural Information Processing Systems, volume 36, pp. 8406-8441, 2023.
13

---

Published as a conference paper at ICLR 2025
Min Wei. Jingkai Zhou, Junyao Sun, and Xuesong Zhang. Adversarial Score Distillation: When score distillation meets GAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the Generative Learning Trilemma with Denoising Diffusion GANs. In International Conference on Learning Representations, 2021.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open- domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
Yanwu Xu, Yang Zhao, Zhisheng Xiao, and Tingho Hou. Ufogen: You forward once large scale text-to-image generation via diffusion gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Yilun Xu, Shangyuan Tong, and Tommi S Jaakkola. Stable Target Field for Reduced Variance Score Estimation in Diffusion Models. In International Conference on Learning Representations, 2022.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, pp. 1-39, 2022.
Semmao Ye and Fei Liu. arXiv:2309. / 1043, 2023.
Score Mismatching for Generative Modeling. arXiv preprint
Mingxuan Yi, Zhanxing Zhu, and Song Liu. MonoFlow: Rethinking divergence GANs via the perspective of Wasserstein gradient flows. In International Conference on Machine Learning, pp- 49984-40000, 2023.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang, Eli Shechtman, Fredo Durand, and William T Freeman. Improved Distribution Matching Distillation for Fast Image Synthesis. arXiv preprint arXiv:2405.1488 2024a.
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6613-6623, 2024b.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015.
Dan Zhang, Jingjing Wang, and Feng Luo. Directly Denoising Diffusion Model. arXiv preprint arXiv:2405. 13540 2024.
Dewei Zhou, Zongxin Yang. and Yi Yang. Pyramid Diffusion Models For Low-light Image En- hancement. In International Joint Conference on Artificial Intelligence, pp. 1795-1803, 2023.
Mingyuan Zhou, Huangjie Zheng. Zhendong Wang, Mingzhang Yin, and Hai Huang, Score identity distillation: Exponentially fast distillation pretrained diffusion models for one-step generation. In International Conference on Machine Learning, 2024.
Yuanzhi Zhu, Xingchao Liu, and Qiang Liu. SlimFlow: Training Smaller One-Step Diffusion Mod- els with Rectified Flow, arXiv preprint arXiv:2407.12718, 2024.
14

---

Published as a conference paper at ICLR 2025
Table 5: The performance of DisBack the distillation from pre-trained EDM model to FastGAN
on FFHQ, AFHOv2, and CelebA in the resolution of 64 x 64.
FFHQ
AFHQv2
CelebA
Model
NFE (1)
FID (L)
IS (T)
FID (4)
IS (1)
FID (L)
(t)
FastGAN LLu et al.
2020
30.27
2.37
28.59
5.94
29.35
2.36
EDM (NFE [1] Karras ct al.
2022
11
29.28
2.97
13.67
10.86
23.05
3.01
Score GAN Francesch et al.
2023
43.89
2.13
53.86
2.13
50.41
2.12
DisBack
23.84
3.27
18.95
7.00
23.16
3.02
A MORE EXPERIMENT RESULTS
A.1 EXPERIMENT RESULTS TEXT-TO-IMAGE GENERATION
We conducted the experiment on LCM-LoRA (uoo et 2023b). LCM-LoRA is a Row- Adaptation (LoRA) version of the Latent Consistency Model (LCM) Luo
DisBack
100
Diff Instruct
et al. (023a)) applicable across fine-tuned Stable Diffu- sion models high-quality, single-step or few-step gen- the
90
eration. In this experiment, student generator and Stable Diffusion v1.5 as the teacher model. We observed that the score distillation under- performs when LCM-LoRA serves as the teacher model. This issue likely stems from the infeasibility of directly converting the outputs of LCM-LoRA into scores.
LCM-LoRA
FID Score
80
70
60
50
1.36x
We distill the LCM-LORA with the proposed DisBack and evaluate the FID scores on the COCO 2014 dataset (Radford al. 202I) with the resolution of 512x512. 12. 50,000 real images and 30,000 generated images were used calculate FID scores. The 30,000 generated im- ages were obtained by generating one image for each of the 30,000 distinct prompts. In the case one-step gener- ation, the original LCM-LoRA has an FID score of 78.26, while the DisBack achieves an FID of 36.37. The change in FID scores over training steps is illustrated in Fig. 7 showing that DisBack achieves 1.5 times acceleration in convergence speed and yields superior generation per- formance within the same training period.
300
600
Epochs
Figure The FID scores of LCM- LoRA distilled from SD1.5 across train- ing steps. The first 40 epochs fer to the computational overhead of the degradation recording stage of Dis- Back. DisBack achieves faster conver- gence and better performance.
A.2 ADDITIONAL EXPERIMENT RESULTS
To explore the distillation performance of the proposed method when the architectures the teacher and student models differ, we opt for EDM Karras et al. (2022) as the pre-trained diffusion model, and FastGAN Liu et al. (2020) architecture for the student model to conduct the experiment. Table shows the performance of baselines and the proposed models the distillation task from diffusion model to a generator. Compared to the original FastGAN, DisBack can effectively improve the generation quality. The results also show that the one-step sampling performance of DisBack better than Score GAN and EDM with 11 NFEs.
A.3 VISUALIZATION OF INTERMEDIATE TEACHER TRAJECTORY
To further demonstrate the effectiveness of the degradation path, we visualized the images generated by the initial generator, the intermediate checkpoints and the teacher model, along the degradation path. As shown in Fig. 8 and 9 We observe that the images generated by the first node in the trajectory are similar to those of the initial generator, while the images generated by the last node in the trajectory are close those of the teacher model. This is consistent with our theoretical analysis.
15

---

Published as a conference paper at ICLR 2025
Gstu
S0
S0s
2
50
S0c
Figure 8: Samples from the initial generator, intermediate teacher trajectory nodes. Here s/ is the teacher model. The teacher model is the pre-trained EDM model on the AFHQv2 dataset, the student generator FastGAN.
G
SA4
$Ø
2
$02
Figure 9: Samples from the initial generator, intermediate teacher trajectory nodes. Here s/ is the teacher model. The teacher model and the student generator are both the pre-trained EDM model on the ImageNet dataset.
B IMPLEMENTATION DETAILS
B.1 DATASET SETUP
We experiment the following datasets:
The FFHQ (Flickr-Faces-HO)) dataset (Karras et al. 2019) is a high-resolution dataset of human face images used for face generation tasks. It includes high-definition face images of various ages, genders, skin tones, and expressions from the Flickr platform. This dataset is commonly employed train large-scale generative models. In this paper, we utilize derivative dataset of the FFHQ called FFHQ64, which involves downsampling the images from the original FFHQ dataset to resolution of 64x64.
The AFHQv2 (Animal Faces-HQ) HQ) dataset (Choi et al. 2020) comprises 15,000 high-definition ani- mal face images with a resolution 512x512, including 5,000 images each for cats, dogs, and wild animals. AFHQv2 is commonly employed in tasks such as image-to-image translation and image generation. Similar the FFHQ dataset, we downscale the original AFHQv2 dataset a resolution of 64 for the experiment.
The ImageNet dataset (Deng al. 2009) was established large-scale image dataset to facilitate the development of computer vision technologies. This dataset comprises over 14,197, 122 images spanning more than 20,000 categories, indexed by 21,841 Synsets. this paper, we use the Ima- geNet64 dataset, subsampled version of the ImageNet dataset. The Imagenet64 dataset consists of a vast collection of images with resolution of 64 64, containing 1,281,167 training samples, 50,000 testing samples, and 1,000 labels.
The LSUN (Large Scale Scene Understanding) dataset Yu et al. 2015) a large-scale dataset for scene understanding in visual tasks within deep learning. Encompassing numerous indoor scene im- ages, it spans various scenes and perspectives. The LSUN dataset comprises multiple sub-datasets, in this study, the LSUN Cat and Bedroom sub-datasets with resolution of 256x256
16

---

Published as a conference paper at ICLR 2025
B.2 EXPERIMENT SETUP
For experiments on FFHQ 64x64, AFHQv2 64x64, and ImageNet 64x64 datasets, the pre-trained models are provided by the official release of EDM Karras et al. 2022). We use Adam optimizers to train the student generator G and 86. with both learning rates set to le The training consisted of 50,000 iterations on four NVIDIA 3090 GPUs, and the batch size per GPU is set to 8. The training ratio between 86 and G remains at 1 : 1. In the Degradation stage, we trained for 200 epochs total, saving a checkpoint every 50 epochs, resulting in total of intermediate nodes along the degradation path {s/g j = 0. 1, 2, 3, 4). In the Distribution Backtracking stage, when 2 3, each checkpoint was trained for 1,000 steps. When < 3, each checkpoint was trained for 10,000 steps. The remaining steps were used to distill the original teacher model %/1''
For experiments on LSUN bedroom and LSUN cat datasets, the pre-trained EDM models are pro- vided by the official release of Consistency Model Song et al. (2023). During the training, we set mma to 80 and keep it constant during the single-step generation process. We use SGD and AdamW optimizers during training to train the generator G and sg, with learning rates set to 1e-3 and 1e respectively. The training consisted of 10,000 iterations on one NVIDIA A100 GPU, and the batch size per GPU is set to 2. The training ratio between 8, and G remains at 4 1. In the Degradation stage, we trained for 200 epochs total and saved the checkpoint every 50 epochs, resulting in a total of 5 intermediate nodes along the degradation path {s/
33
0, 2, 3, 43. In the Distribution Back- tracking stage, when i 2 3, each checkpoint was trained for 500 steps. When i < 3, each checkpoint was trained for 1000 steps. The remaining steps were used to distill the original teacher model s/6.
When distilling the SDXL model, the teacher model and the student generator are both initialed by the pre-trained SDXL model on the huggingface (model id suabiityailstable-dfffuiisoo--- 1.0".. We use Adam optimizers to train G and Sø, with learning rates set to 1e-3 and 1e-2 respectively. The training consisted of 50,000 iterations on one NVIDIA 1100 GPU, and the batch size per GPU is set to 1. The training ratio between and G remains at 1. The training prompts are obtained from LAION-Aesthetics. In the Degradation stage, we trained for 1,000 epochs total and saved the checkpoint every 100 epochs, resulting in total of 10 intermediate nodes along the degradation path {sg, j = 0, 1, 1.... 9}. In the Distribution Backtracking stage, each checkpoint was trained for 1,000 steps. The remaining steps were used to distill the original teacher model s/..
B.3 USER STUDY SETUP
Firstly, we randomly selected 128 prompts from the LAION-Aestheties (Schuhmann et al.) 2022).
Then we use the original SDXL model and the distilled SDXL model to generate 128 pairs of images. Subsequently, the fidelity, detail, and vividness of these pairwise images. 10 volunteers included 6 males and females, aged between 24 and 29. 5 of them have artificial intelligence other 5 of them have other majors. They were given unlimited time for the experiment, and all of the volunteers completed the assessment with an average time of 30 minutes. Finally, we took the average of the evaluation results of 10 volunteers as the final user study result.
randomly recruit 10 volunteers, instructing each to individually evaluate
related majors and the
C THEORETICAL DEMONSTRATION
C. KL DIVERGENCE OF DISBACK
As the KL divergence follows
DKL(q |l p) = E.g log 9
(10)
p
The KL divergence of generated distribution and training distribution at timestep can be written as DKL (1. (xx)
(zx:)
E Ea (m)) log of (xx))
lt (xx)
(11)
Eno))() [log of (xx) log ge (2x)
Eg [log of (x:a) - log (x:))
17

---

Published as a conference paper at ICLR 2025
Thus, the gradient of KL divergence can be estimated
DKL (ge (xx)) 11 (ax)) rE (s6 (2es, t) (æt, t]]
(12)
C.2 STABLE TARGET FIELD
Given Ey do is the training data, at p(a; (b)) is the disturbed data, Xu et al. (Xu et al.
2022)
presents an estimation of the score as:
log (x;;) (xs))( Ezo V p(xx xo) Pr(x,)
(13)
Pt (z,)
The transition kernel p(æ; | æo) follows the Gaussian distribution p(æ; xo) (Mr, I) Here
(4: zg in Variance Exploding SDE (Song et al. (021b) but is defined differently other diffusion models.
1
(xx me)
p xo)
exp(
(14)
((2
20
V P(pp ao))
exp( (xt 1e)=((00 pt))
V..
V (2mm* Jo
20
(15)
|x0) Va. (- (It -Mb) HT) (xx; Ms)
p(
207
Mr at
p(az ar))
o²
Combine Eq. 13 to Eq.
15
we have
V Ve, log pr (xx) = Ex.
p(a:
1t at
Em, Empp(ar
x
11 at
(16)
(r(
(xx)
o²
Let B set of reference samples for Monte Carlo estimation, we have
pr( Ex,, mo)
Σ
p(a,
(17)
20 B
Combine the Eq. (6) and Eq. 17 we can get
V æ, (xz)) Exz
p(x: 2o) (x,)
Σ
p(æ,
|
x
(6)
Me
(18)
of
Pr (xx,)
B
of
2 C
Here the " represents the Monte Carlo estimate.
Depending on the network prediction, the diffusion model can be divided into different types, includ-
ing é prediction (Karras et al. 2022) and xn prediction (Song et al.) 2021a Ho et al. 2020 Nichol
& Dhariwal 2021). When the score Va, (ogee( is estimated by Eq (88 be converted to
20 and 0 by series of transformations.
ē - (z.))(
(19)
to R Va, log of 27,
(20)
D DISCUSSION
D.1 THE GRADIENT ORIENTATION OF SCORE DISTILLATION
18

---

Published as a conference paper at ICLR 2025
To better illustrate the process of training a generator with score dis- tillation, we provide more intuitive explanation. As shown in Figure [0] given noisy data Dt, the direc- tion of 56 (æ,,t) points to the gener- ated distribution and the direction of so (Iz, t) points to the training distri- bution. In Sec[E.3] we used 2D data to illustrate the gradient directions of at on the teacher model 8p and the auxiliary diffusion model 8g. Thus,
Add
Noise
20
21
s so( t)
oo(ore.
90 Gonerated Distribution
4n Training Distribution
Gaussian Noise
sp(2,, b) 20( Back Propagotion
the direction of sy (æt, t) - s (a,, t) is from the generated distribution to- wards the training distribution. Ide- ally, if the predicted score of & and se are both correct, this gradient leads G to update in the right direction.
Figure 10: An illustration of the generator update by the minimization in Eq. (6).
D.2 TWo TYPES OF MISMATCH ISSUE
We find that there are two types of mismatch issues during the training process: (1) The mismatch issue between the initially generated samples and 8g. (2) The mismatch issue between the initially generated samples and sp. For the first type of mismatch issue, it prevents 8, from accurately de- scribing the distribution of the generator, leading to unreliable network predictions and introducing biases when computing the generator's gradients. The combined effect of these two mismatch issues exacerbates the negative impact on the optimization of the generator. Fortunately, among the three initialization methods of 8 proposed in Section 3] when Sy is initialized based on the generated images, its distribution becomes close to the initially generated distribution, which resolves the first type of mismatch issue. In this paper, directly use the degraded teacher model s'y, to initialize Sg. Because the degraded teacher model's distribution closer to the initial generator's 's distribution, S6 initialized by sax can approximate the initial generation distribution and accurately predict the scores of the generated samples from the beginning. However, since šg remains fixed during the training, the second type of mismatch issue cannot be avoided through initialization. Consequently, in this paper, we mainly focus on discussing the second type of mismatch issue.
D.3 TRAINING EFFICIENCY OF DISBACK
While DisBack involves an iterative optimization process during training, the optimization objective of s (Iz, t) aims to minimize the loss of the standard diffusion model based on Eq. 21I, and the objective of student generator aims minimize the KL divergence in Eq. (2). These two optimiza- tion processes do not entail adversarial training as in GANs. Consequently. the optimization process tends to be more stable. recent work Monoflow (Yi et al., (023) also discusses in GANs training
vector field is obtained to guide the optimization of the generator, but the vector field derives from the discriminator and the instability is not mitigated.
ZD Et
min E,,e +
s6 (zI,
(21)
of
min E,, DEL (g. (xx) Ige (xx))
(22)
For the DisBack, training the student generator only requires two U-Nets to perform inference and subtraction. Training &a only involves training single U-Net, and gradients do not need to be back- propagated to Gstu. Therefore, these models can be naturally deployed to different devices, making computational resource requirements more distributed. This ease of distribution allows for joint training on computational devices with limited capacity. In contrast, for GANs and VAEs, which require gradient propagation between models (discriminator generator, decoder to encoder), com- putational requirements are more centralized, necessitating the use of a single device or tools like DeepSpeed to manage the workload.
19

---

Published as a conference paper at ICLR 2025
10-1
Goneroted
Gonerated
Mismatch degree
Mismatch degree
10
0.32
0.5
2
10
20
40
0.32
0.5
2
10
20
40
Sigma
Sigma
(a)
(b)
Figure 11: The results of the pre-experiments on mismatch degree. (a) sg and G are both initialized by the pre-trained EDM (Karras et al. 2002)) on FFHQ. (b) so and G are both initialized by the pre-trained EDM on ImageNet. The mismatch degree on the generated data is greater than on the training data, especially when the noise scale is low.
D.4 VECTOR FIELD
In research, each of the estimated score functions s, for ranging from 0 to N, delineates vector field x + 3x We make a strong assumption behind our proposed method that these score functions represent existing non-existent distributions and that they altogether imply transformation path between $8 and the student generator G Nevertheless, a score fundamentally constitutes gradient field, signifying the gradient of the inherent probability density. A vector field gradient field when several conditions are satisfied, including path independence, continuous partial derivatives, and zero curls (Matthews) 1998). The vector field, characterized by the score functions, may not meet these conditions, and thus there is not a potential function or probability density function. Such deficiencies could potentially hinder the successful training of the student generator and introduce unforeseen difficulties in the distillation process. Specifically, in instances where sy does not precisely represent a gradient field, a highly probable scenario considering se is neural network, the samples generated from Sp could encompass failure cases. Although our empirical studies exemplify effectiveness of the proposed DisBack, the detrimental effects the discussed issue remain unclear. We will further explore this issue in our future work.
E ADDITIONAL DETAILS IN PRE- EXPERIMEEEESS
E. DISTRIBUTION MISMATCH ISSUES
Before conducting research, first carried out preliminary experiments demonstrate that the proposed mismatch issue does indeed exist when using the endpoints of pre-trained diffusion models as teacher models. Using the method proposed Eq. 9 we conducted experiments with the pre-trained EDM model on the ImageNet and FFHQ datasets. We calculated the mismatch degree separately the student model's initial generated data and the teacher model's original training data, and the results are shown in Fig. II We can see that, on both datasets, the mismatch degree on the generated data the pre-trained model greater than that on the real training data, especially when the noise scale is small. This aligns with our hypothesis stated in Sec. [ demonstrating that directly using the endpoint pre-trained model the teacher model leads to a distribution mismatch problem and causes the unreliable predictions of the teacher model.
E.2 A TOY EXPERIMENT GAUSSIAN MIXTURE DISTRIBUTION
To validate the feasibility of the proposed DisBack, conduct experiments on two-dimensional Gaussian mixture data. First, we randomly select 10 Gaussian distributions mixed as the training distribution qu- Next, construct ResNet MLP the two-dimensional diffusion model sy and train it using the created mixture Gaussian distribution. Similarly, we construct a simple MLP as the student generator Gstr and train model 86 with the same architecture so using generated data.
20

---

Published as a conference paper at ICLR 2025
-6
(a) epoch=0
(b) epoch=20000
-2
(c) epoch=30000
(d) epoch=40000
-2
(e) epoch=50000
(f) epoch=60000
Figure 12: The distribution of student generator during the training process. Blue points visualize the generated distribution of and the red points visualize the training distribution qo-
21

---

Published as a conference paper at ICLR 2025
(a) The gradient direction of se (at,, ()
(b) The gradient direction of 86 (æt, ()
Figure 13: The gradient direction of (æqe () and se (æt,, () on The points in (a) are sampled from the training distribution and the points in (b) are sampled from the generated distribution.
89 and Sa train the student generator Gstu- During the training process, we visualize the distribution of the student generator and training data to intuitively demonstrate the changes in the student generator distribution under the proposed training framework. The distribu- tion of Gatn during the training process is shown in Figure 12] As training progresses, the generated distribution d initially expands outward and then gradually convergents towards the training distri- bution. The results show that the proposed method for training the student generator is effective.
Therefore,
E.3 GRADIENT ORIENTATION VERIFICATION OF DISBACK
As mentioned in Sec. when updating Gsta using (q. sp (xt, t) provides a gradient towards the training distribution, while 8a (æt; t) provides a gradient toward the generated distribution.
To validate the correctness of these gradient directions, we experiment two-dimensional data. We evenly sample N data points within the range of (z, y) E [-6, as the noisy data æ4. Subsequently, we depict the gradient directions of æ, based Se and sa respectively. As shown in Figure [3] consistent with theoretical derivation, for any given ær, the gradient direction of 8p (æ., () points toward the training distribution, and the magnitude of the gradient decreases the distance to the training distribution decreases. Similarly, for any given T,, the gradient direction of sy (Iz, t) points toward the generated distribution.
ADDITIONAL SAMPLES FROM DISBACK
We provide additional samples from DisBack on FFHQ 64 x 64 (Figure 15
AFHQv2 64 x 64
(Figure 16 ImageNet 64 x 64 (Figure 17p. LSUN Bedroom 256 256 (Figure 18 and LSUN Cat 256 x 256 (Figure 19p.
G ADDITIONAL ABLATION STUDY
We further explored the impact of different numbers of checkpoints along the degradation path and various degradation strategics on the performance DisBack on the ImageNet64 dataset. The results shown in Tab When degrading for 200iterations under default settings, the FID score with = 3 intermediate checkpoints was 5.23, which outperformed the original Diff-instruct's FID of 5.57. The optimal FID of 1.38 achieved with N 5.
However, with N' = 11, the FID score dropped to 8.35. When there are too many checkpoints in the path, the distributions of the checkpoints (e.g., 66 &6 (/.) are very close the distribution
22

---

Published as a conference paper at ICLR 2025
Table 6: The conditional generation performance of DisBack ImageNet 64x64 dataset.
DisBack
FID (4)
N=1
5.96
N=3
4.88
N=5
1.38
N=11
9.15
N I(Degradation iteration =400)
244.72
of the initial generator. Training with these checkpoints provides limited progress for the generator. Furthermore, having too many checkpoints complicates the checkpoint transition scheduler during distillation, making it difficult to manage effectively. This often leads to inefficient updates for the generator across many iterations, wasting time without meaningful improvement. When the degradation iteration count is set to 400, DisBack fails to correctly distill the student generator. This because excessive degradation iterations result in checkpoints near the initial generator on the degradation path being unable to generate sensible samples. Using these checkpoints for distillation provides no reasonable or effective guidance to the generator, ultimately causing it to fail to learn any meaningful information. In summary, when obtaining the degradation path, it is essential not to overly degrade the teacher model. The degraded teacher model's distribution should be close to the initial generator's distribution while still being able produce relatively reasonable samples. Additionally, having too many or too few checkpoints along the path can adversely affect the final performance.
H FAILURE EXAMPLES
Fig. 14 presents several failure cases of DisBack.
In terms FFHQ, AFHQv2, and ImageNet, while these images already capture the features of the corresponding datasets, the generated results lack accurate and clear backgrounds. 'he potential reasons for this include the fact that these datasets primarily focus on learning foreground content, with low requirements for image backgrounds, making the model difficult to clear backgrounds.
As for LSUN Cat and Bedroom, DisBack successfully generates details such as the cat's fur and the bed's texture, but it does not generate the overall shape and the detailed structure. This may be because the model does not capture the overall information of the data, only capturing local content. This issue may stem from the inherent limitations of U-Net, resulting in poor generation of overall structures in rare cases.
In the future, attempts will be made to use more advanced teacher models improve the distillation algorithm to overcome these limitations. Moreover, we will further explore more advanced generator architectures such as StyleGAN Karras et al. 2020 202 achieve higher- -uallty generation.
I ETHICAL STATEMENT
1.1 ETHICAL IMPACT
The potential ethical impact of our work about fairness. As "human face" included as a kind of generated image, our method can be used in face generation tasks. Human-related datasets may have data bias related to fairness issues, such as the bias to gender or skin color. Such bias can be captured by the generative model in the training.
I.2 NOTIFICATION TO HUMAN SUBJECTS
In our user study, we present the notification to subjects to inform the collection and use of data before the experiments.
Dear volunteers, we would like to thank you for supporting our study. We propose the Distribution Backtracking Distillation, which introduces the convergence
23

---

Published as a conference paper at ICLR 2025
AFHQV2
FFHQ
ImageNet
LSUN Cat
LSUN Bedroom
Figure 14: Failure examples.
jectory into the score distillation process to achieve efficient and fast distillation and high-quality single-step generation.
All information about your participation the study will appear in the study record. All information will be processed and stored according the local law and policy on privacy. Your name will not appear in the final report. Only an individual number assigned you mentioned when referring to the data you provided.
We respect your decision whether you want to be volunteer for the study. If you decide to participate in the study, you can sign this informed consent form.
The Institutional Review Board approved the use of users' data of the main authors' affiliation.
24

---

Published as a conference paper at ICLR 2025
Figure 15: Additional Samples form conditional FFHQ 64x64.
25

---

Published as a conference paper at ICLR 2025
Figure 16: Additional Samples form conditional AFHQv2 64x64.
26

---

Published as a conference paper at ICLR 2025
Figure 17: Additional Samples form conditional ImageNet 64x64.
27

---

Published as a conference paper at ICLR 2025
Figure 18: Additional Samples form conditional LSUN bedroom.
28

---

Published as a conference paper at ICLR 2025
ado AO
Figure 19: Additional Samples form conditional LSUN cat.
29


Reviews:
[{"review_id": "DCVZd2SSjx", "reviewer": "Reviewer_yzv7", "summary": "This paper introduces a novel approach to improve diffusion distillation. The key idea is to include a trajectory of teacher distributions for the student to match. This improves convergence and the final quality of the student model.", "questions": "The original motivation suggests that the teacher training trajectory could be used, but \"the convergence trajectory of most teacher models is inaccessible\". It's not clear to me that the training trajectory of the teacher would be useful for distillation, as it may not align well with the student distribution either. Did you explore DisBack using the training trajectory for trained diffusion models?\n\nDid you explore distillation into smaller student models? Mismatched architectures could be a useful application of DisBack too.\n\nDo you have any samples from the models along the intermediate teacher trajectory? Do these produce sensible samples at all?\n\nOverall, I like the proposed DisBack algorithm and feel that it is sufficiently novel and performant to justify publication. I would give the paper a higher score if the authors provided some more experimental investigation into why their method is successful.\n\n\nMinor:\n\nFig 2. includes the same sample twice (top middle, and middle).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a novel technique that is intuitive and is shown to work well. The experiments show a clear improvement in the convergence speed of the DisBack method and the qualitative results show high-quality single-step samples. The authors apply their method to a variety of teacher models over multiple datasets and demonstrate success and failure cases (in the appendix).\n\nThe paper is easy to follow. Technical content is presented clearly.", "weaknesses": "I felt that the paper writing in places suggested more than was provided. For example, the authors claim that they \"identified this issue arises because existing score distillation methods focus on using the endpoint\". However, the authors provide no such identification in their own work. They provided sufficient evidence that utilizing a trajectory of distributions improves the student but this has not been shown to be necessary. There may be alternative approaches that work well while using only the endpoint. This is present elsewhere in the work, e.g. \"the fat convergence speed is because constraining the convergence trajectory of the generator provides a clear optimization direction\", this is a strong technical claim that has not been adequately explored.\n\nThe authors could do more to explain why DisBack is successful. Some theoretical analysis could help to explain the improved convergence speed, or experiments designed to show the convergence more carefully. For instance, I'd be interested to see how quickly the student model converges to each point on the trajectory, and how closely it matches the distribution. Presumably, this behaviour must be better than linear for DisBack to succeed, which is interesting. Overall, I felt that the idea worked well and was intuitive, but I didn't understand why it worked so well after reading the paper.\n\nThe ablation study is minimal (and I would argue does not qualify as an ablation study). The authors only explore DisBack and the original method side-by-side. Instead, they could also investigate the effect of the number of degradation path checkpoints, different student initializations, different degradation schemes (like using the training trajectory), etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1YbjOtxD0y", "reviewer": "Reviewer_vTDX", "summary": "In this paper, a novel approach is proposed to further improve existing diffusion distillation methods. The proposed approach leverages the convergence trajectory from teacher model to the initial state of student model to guide the training of student model backwards, which mitigates the score mismatching problem at the beginning of the distillation. Empirical results have demonstrated the superior performance of the proposed approach.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper has a clear motivation: to address the initial score mismatch problem in diffusion distillation. Specifically, the authors first identify the suboptimal performance of existing diffusion distillation methods as being due to the score mismatch issue at the beginning of student model training, and then propose a novel approach to resolve it. \n2. The proposed approach is intuitive and effective. It makes sense to follow the degradation path from the teacher model to the student model in reverse during distillation, providing a progressive learning signal for the student and mitigating the initial score mismatch problem. In practice, by introducing this backtracking strategy, the distillation process is shown to be significantly faster than its variant without this technique. \n3. The proposed approach is versatile, as it is orthogonal to other diffusion distillation approaches.", "weaknesses": "1. The state-of-the-art claim for the proposed method is misleading. According to the experimental setup in Appendix B.2, the proposed method trains for 500,000 **epochs** ($\\approx500K \\times \\frac{|D|}{|B|}$ **iterations or steps**, where $|D|$ is the data size and $|B|$ is the batch size). This number is significantly higher than for the baselines. For example, Diff-Instruct only trains for ${\\color{red}50K}$ **iterations** on ImageNet $64\\times 64$, while DisBack (this paper) uses about ${\\color{red}500K\\times 40K=20G}$ **iterations** ($|D|=1,281,167$ and $|B|=32$), which is approximately $40,0000$ times larger. Even if \"epochs\" actually refers to steps (if it is a typo), it still represents 10 times the training length compared with the Diff-Instruct baseline. Additionally, the result (${\\color{green}1.51}$) of DMD2 on ImageNet $64\\times 64$ is achieved by training for ${\\color{red}200K}$ **iterations**. With the extended training setup (${550K}$ **iterations** in total), DMD2 could achieve an FID of ${\\color{green}1.28}$, which is lower than DisBack's ${\\color{green}1.38}$. This raises concerns that the proposed strategy may not fully account for the state-of-the-art performance showcased. This is also supported by their ablation study and Table 3. The variant (\"w/o convergence trajectory\") is essentially Diff-Instruct, as noted in the main text on Lines 391-392. However, even this variant, when trained under the same setting, shows better performance on FFHQ (12.26) versus the original Diff-Instruct (19.93).\n\n2. The speedup shown in Figure 1 is only plotted for epochs 0 to 2000, which covers only the early stage of the distillation. More epochs, either until convergence or until training budgets are exhausted, are needed to better understand how the backtracking strategy behaves throughout training.\n\n3. Although the entire concept revolves around backtracking the degradation path, in actual training, each intermediate checkpoint is only trained for as few as $1000$ steps (for FFHQ, AFHQv2, and ImageNet at $64\\times64$ resolution), while the remaining steps are trained with the original teacher model. This means that the proposed backtracking is used for only a small fraction of the student model's training, which makes it even harder to attribute the superior performance to the proposed strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1XfdkxTpgG", "reviewer": "Reviewer_JLT9", "summary": "This paper propose a DisBack, which is a new distillation method of diffusion models. On the top of the Diff-Instruct, this DisBack propose a better training algorithm. While Diff-Instruct only use the pre-trained diffusion teacher, DisBack makes a series of degraded teachers, and use that teachers iteratively. This makes the student models easy to learn a teacher distribution.", "questions": "My major worry is the reliability of performance and the laborious algorithm. Please respond to my worries.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea is simple but effective. It is makes sense that distilled from degraded teacher make the distillation faster.\n- Degradation Recording algorithm looks reasonable. The degraded teacher finally converges at the initialized student distribution, which make the student easy to learn in the early stage.\n- The result compared to Diff-instruct seems the algorithm is effective.", "weaknesses": "- My major worry is \bthat I can not trust the performance. The paper distilled from EDM in ImageNet 64 which have 2.44 FID, but the distilled student has the performance of 1.38 FID. In my understanding, I think the upper bound performance of the student is teacher.\n\n- I also can not believe user preference study compared to the SDXL teacher. How can it better than teacher? \n\n- The ablation on the number of degraded teacher N is missing. I want to see progressive performance boosting from N=1 (equivalent to the Diff-Instruct) to N= large.\n\n- Is there any scheduling algorithm that changes the teacher in stage 2? It may requires lots of trials to find the schedule that determine when to change that target teacher from degraded to the original.\n\n- Figure 1 is a little bit over-claimed. This algorithm should contains the training costs of stage 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TtTnEt216e", "reviewer": "Reviewer_HUTP", "summary": "This paper introduces Distribution Backtracking Distillation (DisBack), a method to accelerate sampling in diffusion models by addressing the “score mismatch” issue common in traditional score distillation approaches. Unlike existing methods that rely solely on the endpoint of a pre-trained teacher model, DisBack captures the full convergence path between the teacher and student models. It does this through two stages: Degradation Recording, which records a degradation path from the teacher to the untrained student model, and Distribution Backtracking, where the student generator retraces this path to improve alignment with the teacher model.", "questions": "- The score estimation (7) is not general for all noising methods. For example, the score estimation of ddpm has a mean scale $\\alpha_t$. When do distillation, should the teacher and student noising methods keep consistent?\n- Compared with Diff-Instruct which only training student models to fit one teacher model, Algorithm 2 needs to fit $N-1$ intermediate checkpoints, what about the training overhead of this part? In Fig.1, did the epochs for DisBack in x-axis record from the degradation recording stage or from which other points?\n- Any experiments to show the influence of the number of degradation checkpoints and the number of degradation epochs? Will more checkpoints and epochs mitigate the mismatch better?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. DisBack addresses the common “score mismatch” issue in score distillation by incorporating the entire convergence trajectory. DisBack enables the student generator to align more accurately with the teacher model, leading to faster convergence and better optimization paths.\n2. DisBack is designed to be easily integrated into current distillation frameworks, providing a versatile tool to further boost performance in generative model distillation.", "weaknesses": "**Major**:\n- While the authors claim DisBack is orthogonal to those of other distillation methods, there is no evidence to support this point. It would be valuable if the authors could provide further experiments to show it can be incorporated into other distillation methods, like consistency distillation or adversarial score distillation. \n- The paper aims to mitigate the score mismatch issue by employing degradation recording as convergence trajectory for distillation. The mismatch between the predicted score of generated samples and the model's prediction will be degraded but the mismatch between the model's prediction and the teacher's score prediction will be larger in degradation path.  This suggests a potential tradeoff between these two types of mismatches, which could impact the final model’s performance. Providing further analysis or empirical results on this point would strengthen the motivation and effectiveness of this approach.\n\n**Minor**:\n- In Eq.(6), $\\partial x_t/\\partial \\eta$ should be included in expectation, same as (8). \n- Better to use bold $\\epsilon$ for noise and show the relationship between $\\epsilon$ and $x_t$. \n- In Algorithm 1&2, since the loss includes the expectation w.r.t. $t$ and $\\epsilon$, the line to calculate $x_t = x_0 + \\sigma_t \\epsilon$ is unnecessary and misleading. \n- Labels in Fig.7 are wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)