Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models
Authors: Weijian Luo
Tianyang Hu
Shifeng Zhang
Jiacheng Sun
Zhenguo Li
Zhihua Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion model
data-free distillation
implicit generator
knowledge transfer
TL;DR: We propose a general framework called Diff-Instruct to utilize pre-trained diffusion models to instruct the training of arbitrary generative models with a rigorous theoretical derivation and strong empirical results.

Abstract: Due to the ease of training, ability to scale, and high sample quality, diffusion models (DMs) have become the preferred option for generative modeling, with numerous pre-trained models available for a wide variety of datasets. Containing intricate information about data distributions, pre-trained DMs are valuable assets for downstream applications. In this work, we consider learning from pre-trained DMs and transferring their knowledge to other generative models in a data-free fashion. Specifically, we propose a general framework called Diff-Instruct to instruct the training of arbitrary generative models as long as the generated samples are differentiable with respect to the model parameters. Our proposed Diff-Instruct is built on a rigorous mathematical foundation where the instruction process directly corresponds to minimizing a novel divergence we call Integral Kullback-Leibler (IKL) divergence. IKL is tailored for DMs by calculating the integral of the KL divergence along a diffusion process, which we show to be more robust in comparing distributions with misaligned supports. We also reveal non-trivial connections of our method to existing works such as DreamFusion \citep{poole2022dreamfusion}, and generative adversarial training. To demonstrate the effectiveness and universality of Diff-Instruct, we consider two scenarios: distilling pre-trained diffusion models and refining existing GAN models. The experiments on distilling pre-trained diffusion models show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. The experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models across various settings. Our official code is released through \url{https://github.com/pkulwj1994/diff_instruct}.

Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models
Weijian Luo', ;'inyyag Hu Shifeng Zhang", .Jacheng Sun?, Zhenguo Li², Zhihua Zhang'
1 Peking University, 2 Huuawei Noah's Ark Lab
Abstract
Due to the ease of training,, ability to scale, and high sample quality, diffusion models (DMs) have become the preferred option for generative modeling, with numerous pre-trained models available for a wide variety of datasets. Containing intricate infor- mation about data distributions, pre-trained DMs valuable assets for downstream applications. In this work, we consider learning from pre-trained DMs and transfer- ring their knowledge to other generative models a data-free fashion. Specifically, we propose a general framework called Diff- Instruttt to instruct the training of arbitrary generative models long as the generated samples are differentiable with respect to the model parameters. Our proposed Diff-Instrus Instruct is built on rigorous mathematical foundation where the instruction process directly corresponds minimizing novel divergence we call Integral Kullback-Leeie IKL tailored for DMs by calculating the integral of the KL divergence along a diffusion process, which we show to be more robust in comparing distributions with misaligned supports. We also reveal non-trivial connections our method to existing works such as DreamFusion [54). and generative adversarial training. To demon- strate the effectiveness and universality Diff- Instruct,, we consider two scenarios: distilling pre- rre-trained diffusion models and refining existing GAN models. The experiments on distilling pre-trained diffusion models show that Diff-Instruct results in state-of-the- -tt-hhttt single-step diffusion -based models. The experiments GAN models show that the Diff-Instruc can consistently improve the pre-trained generators of GAN models across various settings. Our official code is released through httpt:htpt::tttt thgg....bb..... htt.:p/ss//gghuu....oo////////// cwm.pkulwjw999//di...... httpuu/wwwi99/iiff.ffssttttte
(IKL) divergence.
refining
Introduction
Over the last decade, the field of deep generative models has made significant strides across various domains such as data generation [0) 32 51 [33 24 54 24) density estimation [35 7
image-editing 46, 8 and others. Notably, recent advancmeents in text-driven high-resolution image generation 660 59 58 have pushed the limits of using generative models for Artificial Intelligence Generated Content (AIGC). Behind the empirical success are fruitful developments of a wide variety of deep generative models, among which, diffusion models (DMs) are the most prominent. DMs leverage the diffusion processes and model the data across a wide spectrum of noise levels. Their ease of training, ability to scale, and high sample quality have made DMs the preferred option for generative modeling, with numerous pre-trained models available for wide variety of datasets and applications.
The trained DMs contain intricate information about the data distribution, making them valuable assets for downstream applications.
"Email: luoweijian @stu,,kuu skt..pk..edu.cn. edu.en.
"ooresponndigg to: Tianyang (hutianyangl @ huawei.com)
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

Compared with training from scratch, extracting knowledge from zoo of pre-trained models enables us to learn more efficiently. For instance, [[]] employed variety of pre- pre-trained feature extractors and significantly boosted the state-of-the attt performacce on domain generalization benchmarks. 52 16 exploited the rich multi-modal information stored off-the-shell CLIP models (55) for efficient text- guided image generation. Currently, we are witnessing a rising trend of learning, from models, especially when accessing large amounts of high-quality data is difficult. Such a model-driver learning scheme can be particularly appealing for handling new tasks by providing a solid base model, which can be further improved by additional training data, While this research direction has been extensively investigated for discriminative models and supervised learning tasks 771 38 3 72 55 [6). its application generative models remains largely unexplored. To this end, we are motivated to study the following question.
(Q1): Can we transfer knowledge from pre-trained DMs to other generative models instead of learning from original training data?
The seminal work DreamFusion
S4 demonstrated the feasibility of such a quest for text-to-3I)
generation. Without large text-labeled 3D dataset, [54] took advantage of the rich text-to-image knowledge stored in a large-scale diffusion model, the Imagen [23], to learn the 3D Neural Radiance Fields (NeRF) and achieved surprisingly good performance without using any 3D data. DreamFusion is a rare success and for general scenarios, the task can be difficult due to the vast difference among generative models. DMs represent a class of explicit generative models wherein the data's 's score function is modeled. Conversely, in various downstream applications, implicit generative models are favored due to their inherent flexibility and efficiency. An implicit model typically learns a neural transformation (i.e., a generator) that maps from a latent space to the data space, such as in generative adversarial networks (GANs), thereby enabling expeditious generation.
By exploring diverse architectural designs and latent space configurations, implicit models can readily adapt structural constraints (e.g., molecules must be chemically valid, etc. [14] TO 63)), ), assimilate prior knowledge [3 76 45), and exhibit other advantageous properties, For implicit models that lack explicit score information, how to receive supervision from DM's s multi-level score network is technically challenging, which greatly limits the potential use cases of pre-trained DMs. Therefore, we would like to further address the following question:
(Q2): Can we tackle this challenge so that knowledge from DMs can be more broadly transferred?
In this work, we give affirmative answers to (Q2) and propose a universal framework, Diff-Instruc (DI), to leverage pre-trained DMs to instruct the training of arbitrary implicit generative models as long as the generated samples from the implicit model are differentiable with respect to model parameters. When applied to single-step generation models such as GANs, Diff-Instruct provides an alternative non-adversarial training scheme. When the student model is U-Net (with a fixed time), our method
enters as a strong contender in the diffusion distillation literature [43 61 66), providing extreme acceleration for sampling from DMs with even one single step.
is built on rigorous mathematical foundation where the instruction process directly corresponds to minimizing novel divergence we call Integral Kullback-Leein Leibler (IKL.) divergence. IKL is tailored for DMs by calculating the integral of the KL. divergence along diffusion process, which we show to be more robust in comparing distributions with misaligned supports (Section 3.2)). We also reveal non-trivial connections of our method to existing works such as DreamFusion (Section 3 and generative adversarial training (Section 3.3.2). Interestingly, we show that the SDS objective can be seen as a special case of our Diff-Instruct on the scenario that the generator outputs a Dirac's 's Delta distribution.
Our proposed
(.3ction..3.2)
To demonstrate the effectiveness and universality of Diff- Diff-Instruct, we consider two scenarios mentioned earlier: distilling pre-trained diffusion models to single step (Section[4.) and improving pre-trained GAN generators (Section 4.2) The experiments on distilling pre-trained diffusion models on the ImageNet dataset of a resolution of 64 x 64 show that Diff-Instruce results in state-of-the-a single-step diffusion-baase based models over both diffusion distillation, such as the consistency distillation (66] and
direct training methods [66] 83 73 The experiments on improving GAN generators show that the
Diff- Instruct can consistently improve the tre-trained generators across various settings.
2

---

2
Preliminary
Assume we observe data from the underlying distribution pa(æ)) In generative modeling, we want to generate new samples ~ pp)) where there are mainly two approaches, explicit and implicit. Cur- rently, DMs are the most powerful explicit models while GANs are the most powerful implicit models.
Difffusion models. The forward diffusion process of DM transforms any initial distribution p()) towards some simple noise distribution,
dæ: (dlx
(2.1)
where F' is a pre-defined drift function, G(t) is pre-defined scalar-vale value diffusion coefficient, and w, denotes an independent Wiener process. A multiple-level continuous-indexed score network 8 (x. is usually employed in order to approximate marginal score functions of the forward diffusion process (2.1) The learning of marginal score functions is achieved by minimizing weighted denoising score matching objective [69
/" w(t
LDSM (0)
(()) Br mo mar-ppe (mne (mo) (x),)) V (2g)
(2.2)
Here the weighting function w()) controls the importance of the learning different time levels and denotes the conditional transition of the forward diffusion (2.1). High-quality samples from DM can be drawn by simulating SDE which is implemented by learned score network [65]. However, the simulation of an SDE is significantly slower than that of other models such as implicit models.
Generative adversarial networks. GANs are representative implicit generative models 30, They leverage neural networks (generators) to map an easy-to-sample latent vector generate sample. Therefore they are efficient. However, the training of GANs is challenging, particularly because of the reliance on adversarial training. To train GAN model, a neural discriminator is optimized distinguish the data and generated samples. This leads the creation of surrogate probability metric D2 (->-) between Py (x) and (x)).. The generator is updated based on this metric, with the aim improving the quality of the generated samples 19 of most commonly used GAN [9 can be written as
The objective
Ca = Enpp [log / (x) Ez-p, (1b(g()))(()))) Ly= [log h((()))),
where the training alternates between minimizing Cn and Cg with the other part fixed. For a fixed g. the optimal h should recover the density ratio (pa(x) Px))),, and in turn, Dj is the Jensen-Shannon divergence. There are variants of objectives that minimize other divergences. For instance, if the _(KL) Ex-p, log h(g(=))
1-M((<z)) the objective aims to minimize the KL divergence between generator and data distribution. In Section 3.3.2 we establish the equivalence of our Diff-Instruct with the adversarial training that aims to minimize the KL divergence.
Neural radiance fields. The neural radius field (NcRF) 48
a kind of 3D object model that uses a multi-layer- menti-layer-perceettonn (MLP) to map coordinates of a mesh grid to volume properties such as color and density. Given the camera parameters, a rendering algorithm can output 2D image that view projec- tion of the 3D NeRF. The rendering algorithm is usually differentiable to learnable parameters of NeRF's MLP, this makes the NeRF can be updated through proper instructions on the rendered 2D image.
Diff-Instruct
The main goal of Diff-Instae Diff-Innstrutt is to transfer the knowledge of pre-trained DM other generative models. To demonstrate the universality of our approach, we consider the more general case where the student model is an implicit model, i.e., generator.
Problem setup. Recall our setting that we have pre-trained diffusion model with the multi-level score net denoted as $p.0 (xx) == Væ, (o (xx) where (()) are the underlying distributions diffused at time t according to (2.1] Assume the pre-trained diffusion model provides sufficiently
good approximation of data distribution, i.c., (0) Re pa. For ease of mathematical treatment, use
3

---

Pre-troined Diffusion Model
(x)) " sgl0(z)) ". (x)))
Grad(0)
sØ 0) " (x())) " sp (x 1)
Generator
Xo
XT
An
Forward Diffasion
Geucrated Samppees Xg 8o (X)
Figure 1: Illustration of our Diff- Inftructtttt pipeline. The generator accepts instructions from all diffusion time levels to calculate the gradient of the Integral KL divergence. The gradient is used update its parameters. This data-free free learning scheme enables us to employ pre-trained DMs as teachers to instruct wide variety of generative models.
(0)) pp interchangeably. The goal of our Diff- Iiff-lnsttrct is train implicit model go without any training data, such that the distribution the generated samples, denoted as Pg. matches that of the pre-trained DM. The instruction process involves minimizing certain probability divergences between the implicit distribution and the data distribution.
Instruction criterion. In order to receive supervision from the multi-level score functions 8pe0 (æz), introducing the same diffusion process to the generated samples seems inevitable. Consider diffusing Py along the same forward process as the instructor DM and let g be the corresponding densities at time t. Let sy(0) (æ;) == V (o) (xx) be the marginal score functions. At each time level, how to design the instruction criterion and how to combine different time levels are of critical importance.
To this end, we consider integrating the Kullback-Lein Leibler divergence along the forward diffusion process with proper weighting function, such in (..2). The resulting Integral Kullback-Leiil (IKL) divergence a valid probability divergence with two important properties: 1) IKL is more robust than KL in comparing distributions with misaligned supports: 2) The gradient of IKL. with respect the generator's 's parameters only requires the marginal score functions of the diffusion process, making suitable divergence for incorporating the scoring network of pre-trained diffusion models.
In the following sections, we first formally define the IKL and then go into detail about the mathematical ground of our Diff- Instruct algorithms. We then establish the connections of biff-Instruct to existing methods and discuss in detail a novel application of Diff-Instruct on data-free free diffusion distillation, together with comparison to existing diffusion distillation approaches.
3.1 Integral Sillback--eibble divergence
The IKL is tailored to incorporate knowledge of pre-trained, DMs multiple time levels. generalizes the concept of KL divergence to involve all time levels of the diffusion process.
Definition 3.1 (Integral KI. divergence). Given a diffusion process 2.Tp and a proper weighting
function (t) > 0, [0,7], the IKI. divergence between two distributions p.g is defined as
D 0.77 IKL (q.)) = T
T
w(t)Dxx(g")) (010""ppp)))===="" dt) w(t) mgt [log g (xt) dt, p00)()))))
(3.1)
where (1) and p(t) denote the marginal densities of the diffusion process 2.. time initialized with (0) =q (0) = respectively.
The IKL divergence integrates the KL divergence along a diffusion process, which enables us to amalgamate knowledge from pre- pre-trained DM multiple diffusion times. For simplicity,
the notation D i. (p.q) to represent D)U,.)) (p.q) if the integral exists. Since the KI. divergence well-defined, the proposed IKL divergence, as the integral of KL divergence, is also well-defined
Proposition 3.2. The DIK r. (q:P) satisfies that 1 (g.r) 0, Vg.p. Furthermore, the equality holds if and only if g q= p, almost everywhere under measure p.

---

Algorithm 1: Diff--Instuue Algorithm
Input:
pre-trained DM 8p(t), generator go. prior distribution P2, DM 8g; forward diffusion (2.
while converge do update & using SGD with gradient
Grad (6)
w(t)B xmpp (0
us 00 (8g) (x),) V (xy ( ( 1
0Φ J t=
update 9 using SGD with the gradient
dx (_0( =o(o)))))pp(((a))))) (2re)) 00
Grad(0)= =
w(()B
dl.
an pt deg mp)(s). me (in ( (mg)))))
end return 0,6.
One of the advantages of using IKL instead of KL is its robustness. For instance, with proper weighting function, the IKL well-defined even when the vanilla KL divergence degenerates infinity. This demonstrates that the IKL divergence is more robust than the KL divergence for two distributions with misaligned supports. We consider a famous example in (2 where the generator distribution Pa and target distribution pa have disjoint support. In this case, the KL divergence between Ps and pa degenerates to positive infinity, while the IKL divergence has a finite value for all generator parameters and unique minima that match the generator and data distribution. Check Appendix A.] for details.
3.2 Instruct algorithm
Let ge be the generator of the implicit model. Let ( denote the implicit distribution for samples which are obtained by x6 go (z) (z),z 2 Pz and denote g(1) as the marginal distribution of the forward SDE (2.I) initialized with g()). Let densities and score functions of the pre-trained diffusion model. Our Diff-Instruct aims minimize the IKL between (0) and pl)) so to update the generator's parameters. Following the notations of definition [3.I), we give non-oriin triviali gradient formula for minimizing the IKL in Theorem 3.3 that
and { (e).ee represent the marginal
includes only the score functions.
Theorem 3.3. The gradient of the IKL in (3.1) between q and p!o) is
Oz: dt. 00
Grad(O) 1=0
w(t)E
E. ps (o
( Spco (x:)) my Igg maste//m)))
(3.2)
Theorem 3.3 gives an explicit gradient to minimize the IKL divergence w.r.t the parameter of the generator. Note that the gradient estimation only requires the marginal score functions 8,09 and 8y00- If the marginal score functions of the implicit distribution can be approximated by another diffusion model s tt),,,, we can utilize the gradient formula (3.2) to update the generator's parameter f.
Now we formally propose Diff-Instruct as in Algorithm which trains the implicit model through two alternative phases between learning the marginal score functions 8g, and updating the implicit model with gradient (3.2). The former phase follows the standard DM learning procedure, i.c., minimizing loss function 2.2 with a slight change that the data is generated from the generator. The resulting 8 (Z,,)) provides an estimation of sy (xt). The latter phase updates the generator's ' parameter using gradient from (3.2), where two needed functions are provided by pre-trained DM 8pco (x:) and learned DM & (xx,()) When the algorithm converges, (o( Rs &pt0 (2:)). and the gradient 3.2 approximately zero.
3.3 Connections existing methods
In this section, we establish the connections of Diff-Instruct to two typical methods, the score distillation sampling proposed DreamFusion (4, and the generative adversarial training in Goodfellow al. [9]
5

---

3.3.1 Connection to score distillation sampling
The score distillation sampling (SDS) algorithm was proposed by Poole et al. [54] to distill the knowledge of a large-scale iext-to-image diffusion model into a 3D NeRF model. The idea of SDS has been applied in various contexts, including the text-to-3D NeRF generation based on text-to-2D and the text-guided image editing (20).
diffusion models [54 39
It turns out that the SDS algorithm is a special case of our Diff-Instruct when the generator's output a Dirac's 's Delta distribution with learnable parameters. More precisely, we find that Diff- Instruccss gradient formula (3.2) will degenerate to the gradient formula of SDS under the assumption that the generator outputs a Delta distribution.
Corollary 3.4. If the generator's 's output is a Dirac's ' Delta distribution with learnable parameters, i.c.
((z0)= g())) (zo) Then the gradient formula (3.2) becomes
Grad(O)= /"
dx: dg
-==
(t)
(M (m9)
V Va,logp((2)lzz) Spco (ax1)
dt.
(3.3)
3.3 does not depends on another diffusion model 87(0)) as in (32)) So under the assumption of
Corollary 3.4, there is no need for using another DM to estimate the generator's 's marginal score functions. This is because when the generator outputs Delta distribution, there is randomness in To. So the marginal score functions are only determined by pr (z, zo). The gradient (3.3) is equivalent to the score distillation sampling proposed in DreamFusion [4
The fact that SDS is special case of Diff-Instruct is not a coincidence, as in DreamFusion, the rendered image of NeRF model from certain camera view is a 2D image that is differentiable to NeRF's parameters. Therefore, using SDS learn a NeRF model is essentially an application of using to distill a pre-trained text-to-2D tD diffusion model in order to obtain 3D NeRF object. However, the path we obtain (3.3) is totally different from that in DreamFusion. In DreamFusion, the authors obtained (3.3) by taking the data gradient of the diffusion model's 's loss function (2.2) and empirically omitted the Jacobian term of the pre-trained score network. However, in this work, we first and then specialize it to obtain SDS in natural way.
propose the general formulation of
3.3.2 Connection to GANs
Our proposed Diff-Instruce without integral on time is equivalent to the adversarial training 19 that aims to minimize the KL divergence. Following the same notation as in Section 2 training uses a discriminator h(.) to learn the density ratio to construct the objective for the generator.
the adversarial
Corollary 3.5. If the discriminator h learns the perfect density ratio, i.e.
h(()) Pa(
(2(2))
then updating the generator to minimize the KI. divergence (KT) in Section 2 is equivalent to Diff-Instruct with a weighting function w())= =0 and w(() =, 0.
The Diff- is essentially a different method from adversarial training in three aspects. First, the adversarial training relies on discriminator network to learn the density ratio between the model distribution and data distribution. However, Diff-Instruct employs DMs instead of discriminators to instruct the generator updates. Second, in scenarios where only a pre-trained diffusion model is available without any real data samples, Diff- Instruct distill knowledge from the pre-trained model to the implicit generative model, which is not achievable with adversarial training. Third, Diff-Instructt uses the IKL as the minimization divergence, which overcomes the degeneration problem of the KL divergence via a novel use of diffusion processes and can potentially overcome the drawbacks such mode- drop issues of adversarial training.
3.4 Related works
The pre-trained diffusion models on large-scale datasets contain rich knowledge of the data distribution. There is growing interest in distilling this knowledge to other models 43 that are more sampling-efficien such as implicit generators (50] 26 and neural radiance fields models 48
One significant advantage of our Diff-Instruct framework is its ability to update the generator without real data æ pa. Instead, the knowledge of the data distribution to update the generator is contained
We switch the notation from go (z) to g() since under the assumptions the generator has randomness.

---

in the marginal score function 8,00 as in (3.2) This enables our Diff-Instruct to distill knowledge from pre-trained DMs into flexible generators a data- fata-free manner and provides a major advantage over other diffusion distillation methods that require either the real data or synthetic data from pre-trained DMs. More precisely, There are three kinds of diffusion distillation methods depending on how to use the data for distillation. The first is the data synthetic distillation, which requires using the pre-trained DM to synthesize data from random noises. The student model then learns the mechanism between random noise and synthetic data in order to enhance the efficiency of data generation. Representative methods of data-synthetic distillation are Knowledge Distillation (KD 42) )) Rectified Flow (ReFlow 40]) and DFNO (182). The second method does not require synthetic data from diffusion models but involves real data when distilling. The consistency distillation (CD 666 is a representative method. The third method is pure fata-free distillation, which requires neither real data nor synthetic data. The Diff-Instruct and the Propgressive Distillation (PD [61] are representative pure data-free distillation. Generally, we use the word "data-free distillation" include both data-synthetic distillation and pure data-free distillation. To give a comprehensive comparison of diffusion distillation methods, we summarize three main features of diffusion distillation methods in T The efficiency represents the computational cost of the distillation method. Methods that require synthesizing dataset is inefficient. The data-synthetic distillation requires simulations with pre-trained DMs, thus is inefficient. The flexibility represents whether the distillation approach is capable of distilling knowledge of pre-trained DM flexible generator architectures, for example, the generator whose input and output dimension is different. Diff- Instruct is the only method that can apply to a wide variety of downstream generators.
Furthermore, Diff-Instruct offers very high flexibility to the generator, distinguishing it from traditional diffusion distillation methods that impose striet constraints on the generator selection. For instance, the generator can be a convolutional neural network (CNN)-based or a Transformer-basel image generator such as StyleGAN 28
30 31 37,, or an UNet-based generator (73 adapted from pre-trained diffusion models (32 65 22). The versatility of Diff--Istruct allows it be adapted to different types of generators, expanding its applicability across wide range of generative modeling tasks. In the experiment sections, we show that Diff- is capable of transferring knowledge to generator architectures including both UNet-based and GAN generators respectively. To the best of our knowledge, the Diff-Instruct is the first approach to efficiently enable such a data-free knowledge transfer from diffusion models to generic implicit generators.
4 Experiments
With the abundance of powerful pre-trained DMs with diverse expertise, our proposed Diff-Instruct unlocks their potential as sources of knowledge to instruct a wide variety of models. To demonstrate, we choose the state-of-thhearr DMs from the seminal work by [32 (we denote as EDMs) as instructors and consider transferring their knowledge to implicit generators. In this section, we evaluate the efficacy of DI through two downstream applications: diffusion distillation and improvement of GAN's generator. These two experiments correspond to using UNet and GAN's generator to absorb the knowledge from DMs. In the diffusion distillation experiments, we use Diff-Instruct to distill pre-trained DMs single-step generative models. On the ImageNet 64 x 64 dataset, our Diff-Instruct achieves state-of-the -oe-the--ttttttte performance in terms of FID among all single-step diffusion-basse generative models. Furthermore, in the CAN-immrove experiments, we use Diff-Instruct to improve existing GAN models that are pre-trained to convergence with adversarial training. The Diff-Instruct is shown to be able to consistently enhance the generative performance of pre-trained StyleGAN-2 models on the CIFAR 10 dataset by incorporating knowledge of pre-trained DMs.
4.1 Single-step diffusion distillation
Diffusion distillation a hot research area that aims to accelerate the generation speed of diffusion models. In our experiments, we utilize our Diff- Instructt framework to train single-step generators on CIFAR-10 (6) and ImageNet 64 x 64 [1T] from pre-trained EDM (32 models. We evaluate the performance of the trained generator via Frechet Inception Distance (FID) 21 the lower the better, and Inception Score (IS) (62). ). the higher the better. For additional details about the generator's architecture,
pre-trained models, and the hyper-parameterr parameters on our experimental setup, please refer to Appendix [B.
Performances. Table I and summarize the FID and of the single-step generator that we trained from pre-trained EDMs the CIFAR 10 datasets (unconditional without labels)
with
7

---

Figure 2:
Generated samples from one-step generators that are distilled from pre-trained diffusion models on different datasets. Left: FFHQ-64 (unconditional); Mid: ImageNet-64 (conditional); Right: CIFAR-10 10 (unconditional).
Table 1: Unconditional sample quality on CIFAR-
Class clasicconiiii sample quality on CIFAR-IO and ImageNet 64 x through diffu- sion generations. "Methods that require synthetic data construction for distillation. Methods that
Table 2:
10 through diffusion generations. "Methods that
require synthetic data construction for distillation.
Methods that require real data for distillation.
METHOD
NFB()
FID
require real data for distillation.
Multiple Steps (include Diffusion Distillation)
METHOD
NFE
FID
DDPM 22
1000
3.17
9.46
Multiple Steps (include Diffusion Distillation)
LSGM 68
147
2.10
EDM [2]
35
1.79
PFGM 74
110
2.35
9.68
HDM-Heun
32
20
2.54
EDM 32
1,97
EDM-Buler
20
6.23
EDM-Heun
10
15.56
DDIM
DPM-solver-2
5.28
Single Step
DPM-solver-3 41
6,03
EDM 32]
314.81
3-DEIS 78
10
4.17
Diff-Instruct
4.19
UniPC 80
5.10
UniPC
23.22
Class-conditional ImageNet 64 x 64. 'Distillation techniques.
Denoise Diffusion GAN(1=2) 73
4.08
9.80
METHOD
NFE
FID
[1
9.05
Multiple Steps
8.85
ADM [R]
250
2.07
CD 60
2.93
9.75
SN-DDIM
100
17.53
Single Step
HDM 12
79
2.44
Denoise Diffusion GAN(T=1)
73
14.6
8.93
HDM Henn
12
10
17.25
KD' 5
9.36
GGDM 70
25
18.4
TDPM 83
8.91
8.65
CT 66
2
11.1
1 ReFlow
40
378
1.13
PD
2
8.95
CT 66
8.70
8.49
CD
66
2
4.70
I-RePlow (4distill))
9.08
Single Steps
2-ReFlow (+distill))
40
4.85
9.01
HDM 12
154,78
3-ReFlow (+disiill))
40
5.21
8.79
PD 51
15.39
PD 61
8.34
8.69
CT 66
13.00
CD-L2
66
7.90
CD 1.2 66
12. 12.10
CD-LPIPS 60
3.55
9.48
CD-LPIPS 66
6,20
Diff-Instruct
4.53
9.89
DifT-Instruct
5.57
and the conditional generation on the ImageNet 64 x 64 data. Diff-Instruc performs competitively across all datasets among single-step and multiple-step diffusion-based generative models, which involve both models from diffusion distillation or direct training.
As shown in Table 2 on the ImageNet dataset of the resolution of 64 x 64, Diff-Inntruct outperforms diffusion-based single-step generative models in terms of FID, including both distillation methods that require real data or synthetic data, and even models that are trained from scratch. On the unconditional generation of the CIFAR10 dataset, Diff-Inntruct achieves the state-of of-the-art IS among diffusion- based single-step generative models but achieves the second-bess FID, only worse than the consistency distillation (CD) (66] which requires both real data for distillation and the learned neural image metric (e.g. LPIPS (9]). The conditional generation experiment on the CIFAR10 10 dataset shows
that the Diff-Instruct performs better than 20-NFE diffusion sampling from EDM model [2 with Euler-Maruyama discretization but worse than 20-NEF Heun discretization.

---

205.8
Core stency Distilation (, mIPS
Corts stancy Disiilation (L2)
Cons stency Dictilation (L.)
0
90.0
75.0
35.0
4.6
0.00
0:07
0.14
0.22
0.29
0.36
0.43
0.50
0.57
0.65
0.72
Figure 3: Comparison of FID convergence on diffusion distillation of CIFARIO unconditional EDM
model 2
The FID value of consistency distillation is obtained from its original paper [66].
Figure shows some non-cherry-picked generated samples from a single-step generator trained with Diff-Instruct on the FFHQ [28). ImageNet [II), and the CIFAR 10 (36] datasets of the resolution of 64x64. In conclusion, our Diff-Instruct can achieve competitive distillation performance under the most challenging conditions with no synthetic or real datasets. We put more discussions and analyses in the Appendix B.1
Remark 4.1. In Tad Table 2 PD, CD, and Diff-Instruct all use the EDM teacher, and the same UNet student with the same architecture as the teacher model (The author of consistency distillation re-implemented the PD for EDM, so the reported FID for PD lower than that in PD's original paper). The Diff- Instruct generator uses the same UNet architecture as the teacher diffusion model, so the number of sampling steps (NFE) represents its inference time costs. As we show in the upper part of T sampling from our learned one-step generator with 1 NFE results in an FID of 4.19, which significantly better than its teachers with 10 NFEs with an FID of 15.56. So we conclude that one-step generator trained with biff-Instruct achieves at least 10+ times acceleration (more efficient) than its teacher diffusion model.
for diffusion distillation is the fast convergence speed. We empirically find that Diff- has a much faster convergence speed than other distillation methods and has tolerance for large learning rate for optimization. In Figure 3 we show the convergence of FID with respect to the optimization iterations of the generator trained on an unconditional DM on the CIFAR 10 dataset. We set the optimization step size of Diff-Instruct to be 1e-4 and the FID of the generator trained with Diff--Inttruct converges fast within 7k iterations. However, the FID of the distilled diffusion model with consistency distillation algorithms does not converge with less than 7k iterations. One possible reason for the fast convergence speed is that Diff-Instruct s student model is a one-step generator that does not need to take multiple-tin indexes in contrast student models of other distillation methods such as CD and PD. This makes Inff-unst efficient when distillation without the need for learning at multiple time levels.
Fast convergence speed. Another advantage of applying
4.2 Improving generative adversarial networks
Another application of Diff--Insttut to improve the generator of GAN models that are pre-trained with adversarial training 3.3.2
Experiement settings. We take the pre-trained EDM model [32] the CIFARIO datasets as the instructor and the pre-trained StyleGAN-2 [29] models that are assumed to converge with adversarial training under different settings (conditional or unconditional with different data augmentation strategies). Our goal is to use pre-trained DMs to further improve pre-trained generators. We initialize the generator in the Diff- Instructt algorithm with the pre-trained StyleGAN-2 generator and initialize DMs for implicit distributions with the pre-traines EDM models. We then use Diff- Instructt to update the generator. We put more details in Appendix B.2
Performance. shown in Table 3 and 4 our Diff- Diff-Instruct can consistently improve the pre-trained generator's ' performance in terms of FID. More precisely, the FID of the pre-trained StyleGAN-2 with Adaptive Data Augmentation (ADA) is improved from 2.42 to 2.27 for the conditional setting and

---

Table 3: caas--ooniiional sample quality Table 4: Unconditional sample quality CIFARIO through GAN models. Models that
CIFARIO through GAN models. implemented.
Models that
implemented.
METHOD
FID (
IS (1)
METHOD
FID (1)
IS (1)
BigGAN [5]
14.73
9.22
SNGAN 49
21.70
8.22
BigGAN+Tune
9.07 0.13
ProGAN 17
15.52
8.56 1 0,06
MultiHinge
33
6,40
9.58 0.09
AutoGAN 18
12.42
8.55 0.10
FQ-GAN 81
5.59
8.48 0.099
ANGAN+DGflow
Stylegan2
0.00
TramsGAN 27
9.02
9.26
Stylegan2
7.01
9.207
StykGAN2 30
8.32
9.21 1: 0.09
Stylegan22 DI
6.62
0.00
StyleGAN2I
8.21
9.09 0.09
Stylegan2+ADA
3.49
1.007
StyleGAN2' +DI
7.56
9.16
Stylegan2+ADA+TTnee
29
2.42
10.14 0.09
StylGGAN2+AAA 29
5.33
10.02 + 0.07
Stylegan2+AAA--Tune DI
2.27
10.11
0.10
StyleGAN2+AADA+TTUme 29
2.92
9.83 + 0.04
StyleGAN2+ADDA-Tumee DI
2.71
9.86 0.04
from 2.92 2.71 for the unconditional setting. The experiment shows that biff-unstruct is able to inject the knowledge of pre-trained diffusion models to enhance the generators further.
The results demonstrate that Diff-Instruct a powerful method capable of improving existing GAN models that are supposed to converge with adversarial training. There are two possible reasons for such improvements. First, our Diff-Instruct utilizes well-trained diffusion models supervise the generator. For instance, on the CIFAR 10 dataset with conditional labels, the teacher EDM model can achieve the FID of 1.79, which is significantly better than StyleGAN2 with an FID of 2.42. Second, Diff-Instruct takes diffused data into account when minimizing the IKL, overcoming the potential degeneration issues of the divergences that adversarial training intends minimize.
5
Discussion
This work presents a novel learning paradigm, method that enables knowledge transfer from pre-trained diffusion models into generic generators in data-free manner. The theoretical foundations and practical methods introduced in this work hold promise for advancing the utilization of diffusion generative models and implicit models across various domains and applications.
which is to our best knowledge, the first
Nonetheless, Diff--Intrut has its limitations that call for further research along this line. First, with the abundance of powerful pre-trained DMs with diverse expertise, levering multiple models as instructors another promising direction that not investigated in this work. Second, even though data-free is feature of our method, utilizing real data can potentially boost the learning process. The potential benefits of incorporating both Diff-Instruct and training data have been explored yet. Lastly, the extreme case where we have only data and no pre-trained DMs, our DI framework can still be adapted. Potentially we can train teacher diffusion model with data, and concurrently, use it to instruct the student model. This indirect way of training may enable other generative models such as GANs, to enjoy the benefits of diffusion models, e.g., ease training, ability to scale, and high sample quality, etc.
Acknowledgments and Disclosure of Funding
W. Luo and Z. Zhang have been supported by the Beijing Natural Science Foundation (Z190001) and the National Natural Science Foundation of China (No. 12271011).
10

---

References
[1] Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discriminator gradient flow. In International Conference Learning Reprexentations.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214 223, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL /tcLltttppp/prccdd.nngg mlr. wwwss jovskyyl html
[3] Devansh Arpit. Huan Wang. Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. arXiv preprini arXiv:21 10832, 2021.
[4] Fan Bao, Chongxuan Li, Jiacheng Sun, Zhu, and Bo Zhang. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206. 07309, 2022.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019. URL
net https::/ htwp:rreeeeeeeeettottm.te.eee wttp:/oormmmiiii/ http://ooommmmmdlxssssx3110a
[6] Charlotte Bunne, David Alvarez-Melis, Andreas Krause, and Stefanie Jegelka. Learning generative models across incomparable spaces. In International conference on machine learning, pages 851-861. PMLR, 2019.
[7] Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, pages 9916-9926, 2019.
[8] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: iiffusiin-based semantic image editing with mask guidance. ArXiv, abs/2210.1 114277 2022.
[9] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pages 88-105. Springer, 2022.
[10] Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint 1rXiv: 2018.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages
Icee, 2009.
[12] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[13] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint urXiv:2005.00341 2020.
[14] Luca Di Liello, Pierfrancesco Ardino, Jacopo Gobbi, Paolo Morettin, Stefano Teso, and Andrea Passerini. Efficient generation of structured objects with constrained adversarial networks. Advances in neural information processing systems, 33:14663-14674, 2020.
[15] Qishi Dong, Awais Muhammad, Fengwei Zhou. Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung-Ho Bae, and Zhenguo Li. Zood: Exploiting model zoo for out-of-distribution generalization. Advances Neural Information Processing Systems Volume 35, 2022.
[16] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan-nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics (TOG), 41(4):1-13, 2022.
11

---

[17] Hongchang Gao, Jian Pei, and Heng Huang. Progan: Network embedding via proximity generative adversarial network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1 1308-1316, 2019.
[18] Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3224-3234, 2019.
[19] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[20] Amir Hertz, Kfir Aberman, and Daniel Cohen-Or. Delta denoising score. arXiv preprint vrXiv:2304.07090 2023.
[21] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.
[22] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[23] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao. Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023000 2022.
[24] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458, 2022.
[25] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 3887 PMLR, 2022.
[26] Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. arXiv preprint arXiv: arXiv:1810.080 1eto.03440035 2018.
[27] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34: 14745-14758, 2021.
[28] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4401-4411 2019.
[29] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33. 2020.
[30] Tero Karras, Samuli Laine, Miik Aittala. Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110-8119 80198119,22000011 2020.
[31] Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. Advances in Neural Information Processing Systems, 146852-863, 2021.
[32] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurlPS, 2022.
[33] Ilya Kavalerov, Wojciech Czaja, and Rama Chellappa. A multi-class hinge loss for conditional gans. In Proceedings of the IEFF/CVF winter conference on applications of computer vision, pages 1290-1199, 2021.
12

---

[34] Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: A diffusion model for text-to-speech via classifier guidance. In International Conference on Machine Learning, pages 11119-11133 PMLR, 2022.
[35] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Biaa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 10215-10224, 2018.
[36] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-I0 Dataset. online: http://www. cs. toronto, edu/kriz/cifar. html, 55, 2014.
[37] Kwonjoon Lee, Huiwen Chang, Lu Jiang, Han Zhang, Zhuowen Tu, and Ce Liu. Vitgan: Training gans with vision transformers. XrXiv preprint arXiv:2107.04589 2021.
[38] Ziyue Li, Kan Ren, Xinyang Jiang, Bo Li, Haipeng Zhang, and Dongsheng Li. Domain gener- alization using pretrained models without fine-tuning. arXiv preprint arXiv:2203.0400 2022.
[39] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang. Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung-Yi Yi Lin. Magic3d: High-resolution text-to-3d content creation. ArXiv, wbsp/211.1000000 104440, 2022.
[40] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 2022.
[41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206.0000 00927, 2022.
[42] Eric Luhman and Troy Luhman. Knowledge distillation iterative generative models for improved sampling speed. arXiv preprint
2021.
[43] Weijian Luo. A comprehensive survey on knowledge distillation of diffusion models. arXiv preprint arxiv:2304.002222222 2023.
[44] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. 20/7 IEEE International Conference on Computer Vision (ICCV), pages 2813-2821, 2017.
[45] Yunwei Mao, Qi Hc, and Xuanhe Zhao. Designing complex architectured materials with generative adversarial networks. Science advances, 6(1))caaz41169 2020.
[46] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0107. 2021.
[47] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. ArXiv, abs/2211 11.02200,, 2022.
[48] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[49] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations.
[50] Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv: 161000348, 2016.
[51] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.0967: 2021.
[52] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide; Towards photorealistic image generation and editing with text-guided diffusion models. arXiv 21X: 2021.
13

---

[53] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. arXiv preprini arXiv: (te.ii6090.3999 2016.
[54]
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[56] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Recycling diverse models for out-of-distribution generalization. arXiv preprint arXiv:2212. 10445, 2022.
[57] Alexandre Rame, Matthicu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, patrick gallinari, and Matthicu Cord. Diverse weight averaging for out-of-distribution gen- eralization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL
https://cpenreviers F J_MQUUB
[58]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learming, pages 8821 1821- PMLR, 2021.
[59] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hicrarchical text-conditional image generation with clip latents. arXiv preprint 0rxiv:224.006155 2022.
[60] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. / /487, 2022.
[61] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffu- sion models. In International Conference on Learning Representations, 2022. URL
net wwt. /ttp//mmmma-aaixiiie.
[62] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-22442 2016.
[63] Bidisha Samanta, Abir De, Gourhari Jana, Vicenç Gómez, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Gomez-Rodriguez. Nevae: A deep generative model for molecular graphs. The Journal of Machine Learning Research, 21(1):4556-458 2020.
[64] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint xr11::000.0005000, 2020.
[65] Yang Song, Jascha Sohl-Dicksteen, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2020.
[66] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
[67] Sandeep Subramanian, Sai Rajeswar, Francis Dutil, Chris Pal, and Aaron C. Courville. Adversarial generation of natural language. In ReeAANLL@AAL, 2017.
[68] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances Neural Information Processing Systems, 34:1 1287-11302, 2021.
[69] Pascal Vincent. Connection Between Score Matching and Denoising Autoencoders. Neural 2011.
Computation,
14

---

[70] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2022.
[71] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Kr- ishnamurthy Dj Dvijotham, and Ali Taylan Cemgil. A fine geeegrranndd analysis on distri- bution shift. In International Conference Learning Representations, 2022. URL http::://ppereevee
net htte://ooonnnddDDeeeeeee
[72] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-23993 PMLR, 2022.
[73] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2021.
[74] Yilun Xu, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. Poisson flow gener- ative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL net forum?id=voy TRqcRh
[75] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[76] Li-Chia Yang, Szu-Ya Yu Chou, and Yi-Hsuan Yang. Midinet: A convolutional genertiive network for symbolic-domee domain music generation. arXiv preprint 1rXiv:1703.1084. 2017.
adversarial adversssria
[77] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI Conference Artificial Intelligence, 2017.
[78] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902. 2022.
[79] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
[80] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.048800 2023.
[81] Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization improves gan training, arXiv preprint arXiv:2004.02088. 2020.
[82] Hongkai Zheng, Weili Nic, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211. 131449, 2022.
[83] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. The Eleventh International Conference on Learning Representations, 2023.
In
URL
https: www.:ww..eeeeeeeeeee.eeeeeeeeeee ftr./formmmiii -DD
15

---

A Technical details
A.1 Robustness of Integral KL divergence
One of the benefits of using the Integral KI. divergence over the traditional KI. divergence is its robust- misaligned density support. To illustrate this advantage, consider well-known example in 12 Let be a random variable following the uniform distribution on the unit interval [0,1]. Consider IP be the distribution of (0,z) E R2, Now let g=(=)= (0,z), where 0 is a single real parameter. The den- sity of Po and le are po ()) 2) - ( 11 and po (z) = (x (1)))lo.111 (1]) Since for each 0 + 0, the support of py and Po does not intersect, the KL divergence between Po and Po is ill-defined with
+00 0
0
(KI(PPoPPo)
(A.1)
0=0.
The same is also true for the
divergence where
f1 log22 o
(Ps(Po.Po)=
(A.2)
0
0-0.
So minimizing the KL divergence with gradient-based algorithm does not lead the generator to converge to the correct parameter 0 However, IKI. provides finite and reliable objective for training the generator. More precisely, considering simple diffusion
dæ, dw..
(A.3)
The marginal distribution of (x,z) under diffusion A.3 initialized with and Pa writes
Po (2) (x.2)= N(z;,,t)ss,
p9 (r,2)) / (((ss,t)ds.
which are defined on (ær²) e R2. The notation (x:Hu,o0²) represents the density function of Gaussian distribution with mean y and variance o². The IKI. divergence with weight function wt( thus has the expression
log (2) (xx²)
(PoPPP)
w(t)
E (1) (a.a))
dt
(a.2) (a,a)~m)
Pb (0)
(ar,z)
log (xx00,1
w())
E
(...)) (9) (a.z)
dt
(x;;0,tt_
N (xx0,tt
w(t)
E E log
dt
(A.4)
(zx00tt)
1
w(t)
E Em~N(x:0
[22 - (x -x-
df
2t
1
w(t)
E B2
[20r 07] de
2t
w())
8²
dt AER.
By properly choosing the weighting function wt),, DIKL L finite as long as f w(t) 2t dt is finite, For instance, wt) = 1/t if and w(()= = if I as a simple choice The IKI. divergence (A.4) is differentiable quadratic function of parameter with single minima 0 0 which lead to the (Po 00
Table shows a summary of the comparison among IKL divergence, KL divergence and the Wasserstein distance between Pa and P.. Our IKL is more suitable for learning with gradient- based optimization algorithms.
In practice when distilling from pre-trained diffusion models, we use the same weighting function for training pre-trained diffusion models for Diff- Instruct, which is also inverted U-shaped.
16

---

Table 5: Comparison of divergence property against IKL.
Divergence
distance
smooth
IKL (ours)
0
KI.
+00
X
Wasserstein
X
Lansen-Shannon
log2
(A.2)
X
A.2 Proof of Theorem 3.3
Proof. Recall the definition of g(),, the sample is obtained by ao go(2), Pz. and at pep (D; an) according to forward SDE (2.1). Since the solution of forward, SDE is uniquely determined by the initial point Do and trajectory of Wiener process ()
slightly abuse the notation and let 2; = =F (=). ()) to represent the solution of æ; generated by Zu and We let W[0 1 PP to demonstrate a trajectory from the Wiener process where Fw represents the path measure of Weiner process on E [0, There are two terms that contain the generator's s parameter O. The term 3t contains parameter through æy - go (2),2=~>pp The marginal density g¹)) also contains parameter implicitly since g()) initialized with q (0) which is generated by the generator. To demonstrate the parameter dependence, we may use % to represent (t))
The p( is defined through the pre-trained diffusion models with score functions 8p10- The IKL. divergence between (*) and p(*) is defined with,
D 1o.T (0) IKL (00
(0)) = wt)DDkl(g) (0000)pp000))))00000000)0))0
1=
La (art)
w(t) m a
log
dt
p00) (xx))
4o (1)
(F (F(g(()) dt
w(t)BB smpee
[lo (() (F((())) (z)2))
(A.5)
t=0
Taking the 0 gradient of IKL A.5
we have
00
D
TKL
Io.T
(9g
(8)
(1)
0
T
log 4 (F(10(z)
w(t)B
00
1=0
www
(tt)
[ (z)(
(F(go
(())
log
30
(Fe (z),
(()E
Va, [log
4
(F (g)(()))
OJ Ogs (2),w)
[[go (g)(()) ())
00
+
(()E E
loga (1) (,,=))))FF(
(t) xoOpS wow m000(s)) a: [log 4o (xx) Ox,
+ 1" w(t)lE
wimps" 00 logge (xt)
a
(0)
(x. (x;)) 00
F(og.
A++B
(A.6)
The term A in equation A.6 writes
Ox:
==
w(t) (eggtt) (2xt) 9pcn (2x1)
(A.7)
00
1=0
17

---

We show that the term B in equation A.6 vanishes.
w
0
logge (x.)
(1)
00
/
w(t)
acs )9
(x:e) 00
Po
w(t)
00 90
a
dx:
==0
w())
θ
9o (a. (a.)
(t)
(A.8)
08
0
(1)
1dæ.
00
1=
(A.9)
(t) The equality (A.8) holds if function do (a:) satisfies the conditions (1). (0(( (ar ) is Lebesgue integrable
for æ with each 0: (2). For almost all a E IR P, the partial derivative 04g (1) (a)/00 exists for all 0 e.
(3) there exists integrable function h.):: + R. such that d (ax) (ax) for all its domain.
Then the derivative w.r.t 0 can be exchanged with the integral over æ, i.c.
f
θ (1) 00 (x) 00
0 /
(1) (x)dx.
Remark A.1. In practice, most commonly used forward diffusion processes can be expressed as form of scale and noise addition:
æt = (()ax B((),, e~N(e;
(A.10)
So the term 2 Pes w 2~ e~ (e: a a
Pw, It F(xo, w) in equation A.5 can be instantiated as
A.3 Proof of Corollary3.. 3.4
Proof. Since the q(20)) dg(0) (2o), thus the conditional distribution and marginal distribution coincides, i.c.
(x))) pr (0)) (wo)d - (20 (xo) -p: (x ((()
So the marginal score function writes
Syie) (axx) = v., (O V (ogp(
So the gradient formula 3.2 turns to
Va (zg - -ppc (x:) dx, dt.
Grad( = w(t)E
1=
(m0)))
(9)
00
If we define e-network ep(x, 1) = -8t0 (a.)(/() and consider the forward diffu-
sion (=) = N(x; a(t)æ0, (t) the forward diffusion can be implemented with 2, ( NN(:;
then the objective turns
Grad (SDS) (0) /
"w())
E00 dt, (p(
Bag -oo 21 #
(A.11)
(t)
00
18

---

which recovers the SDS gradient estimation proposed in DreamFusion [54] In summary, we have shown that our Diff-Instruct's gradient is equivalent to the gradient formula of score distillation sampling (SDS) when the generator outputs single data that is differentiable to the generator's parameters. This not coincidence, as NeRF model can viewed as generator that outputs Dirac's Delta distribution when the view direction is fixed. Therefore, using SDS to learn text-conditioned NeRF model essentially an application of using an approximated version of Diff-Instruct distill a pre-trained text-to-2D diffusion model in order to obtain NeRF.
A.4 Proof of Corollary 3.5
Proof. Following the same notation in section
consider the adversarial training that minimizes The learning objective
the KI. divergence as we introduced in Section
2
C L) (0) -1)=EE [log 1 (=)))
(A.12)
(z))
Assume the discriminator h(.) is optimal, then
pa(æ)) and log
1-h(x)
log pa(x))
h(x))
Pp(
h(x))
Py
Then gradient for the generator parameter 8 turn
i (( (0) Ezap. V (log
1 h(())
Ogs (z)
)===)(
08
h(x)
00
Egups V x (log Pa(
Ögo(z)
)=() 00
Py p,(
0go(z)
Egup.
V n logpa(a;)
V wloggp
(==)6(x)
00
Ox
E seps.
V (alogp((x))
V Valog
(=)
00
smpp (=)
- - 8y (x()
(A.13)
00
where sy (xx) = V logpp and (x)
(x) denote the score function of the generator and
the data distribution. The equation (A.13) shows that the adversarial training that minimizes the KL
divergence is equivalent to the gradient formula (3.2) with a special weight function w()) = 1 and
w() > for all ( > 0, if the discriminator can be trained to be optimal.
A.4.1 Comparison of Distillation Methods
Table 6: Comparison of Diffusion Distillation Methods. The term efficiency represents the training effi- ciency of DMs. fleibiiiy means whether the student model needs to have the same in-out dimensions.
Method
data-free
flexibility
efficiency low low low medium low XXX
ReFlow 40
ANKY
DENO 92
KD 42
CD 66
X
PD 61
(ours)
high
B More on experiments
To demonstrate the efficacy of our proposed Disf-Instruet, we choose the state-of-the-art EDMs instructors, which have achieved sate-of-the-an generative performance several benchmarks
19

---

Table 7: Hyperparameters used for Diff-Instruct for Diffusion Distillation
Hyperparameter
CIFAR-IO(Uncoon)
ImageNet 61x61
CIFAR-I0(Cond)
DM
Generator de
DM
Generator dp
DM
Generator
Learning rate
Te-5
le-5
le-5
le-5
Te-5
Te-5
Batch size
64
64
96
96
a(C')
2.5
2.5
5.0
5.0
2.5
Adam 3
0.0
0.0
0.0
0.0
0.0
0.0
Adam 9
0.99
0.99
0.99
0.99
0.99
0.99
Training iterations
100k
100k
50k
50k
100k
100k
Number of GPUs
4
4
8
8
Initial
Early
Middle
Converge
Figure 4: Left: Demonstration of the training process for Diff-Instrus for diffusion distillation. Generated samples with the same latent vectors and different generator weights during training put from the top to the bottom. Right: generated samples from a one-step generative model distilled from pre-trained class-conditions EMD model the CIFAR10 dataset.
such CIFARIO and ImageNet 64 64.
The EDM model depends on the diffusion process
dæ; = G()
(B.1)
Samples from the forward process B. can be generated by adding random noise to the output of generator function, i.e., H, 37 (()) where 6 (O, I) is a Gaussian vector and
(t) =
is a function with explicit expressions. We download the pre-trained model 5 and consider transferring their knowledge to implicit generative
checkpoints from the official website models, specifically UNet and StyleGAN as generators.
We calculate the FID and IS in the same way as the StyleGAN2-ADA codebase. For the ImageNet 64% x 64 dataset, we use the same pre-processing scripts as the EDM model on ImageNet 64 x 64 dataset.
B.1 Detailed experimental settings diffusion distillation
When the downstream generative model is UNet generator, our DI provides another way for diffusion distillation, directly competing with progressive distillation 61 or consistency distillation [66].
We initialize the diffusion model for the implicit distribution 86 with the weight parameters of pre- pre-trained DM. The initialization the generator is put in the following paragraph. With the pre-trained DM, initialized generator, and the initialized DM for implicit distribution, use Instrucst algorithm to update the DM for implicit distribution and the generator's parameters. the same
We the Adam optimizer for both the DM and the generator. For the generator, exponential moving average (EMA) technique as the EDM model's 's training scripts.
To make fair comparison, we do not compare approaches that involve other components such as classifier guidance or additional architectures for the generation as [82 because these models have significantly larger model sizes for inference.
'hups://github com/N https///githbb.om////////ttttttt
wttp:iiihuuucoo///laaaaee Vlabs//dd
20

---

Table 8: Comparison of Memory Costs and Wall-Clock Timing of CD and Diff--nnstuce The Peak GPU-Memo shows the Maximum observed GPU memory caused by the Tensor and Computational Graph of the host GPU. Sec-per-K Iterations report the wall-clock time for each training iteration. For Inff-Instruct, each iteration consists of two stages as Algorithm I the main text. Test environment:
PyTorch 12.1 and Torchvision 0.13.1, and borrc.distrbbuted.arraee
on 2 V100 GPUs.
Method
Peak GPU-Memo(GiB) Peak CPU-Memo
Sec-per-K Iterations
CD
9.55
2.75
0.0489
Instruunt
10.40
2.78
0.0728
Initialization of the generator. When the generator is chosen have the same architecture as the pre-trained diffusion models (i.c. UNet in most cases), we can initialize the generator with a special method with the score network of the pre-trained DM. Taking the forward diffusion process (B.1) as an instance. The time-indexed score functions explicitly define a data- iiata-prediction transform through Tweedie's 's formula
= + V, logg (x:)
(B.2)
This formula tells that the marginal score functions can be converted to data data-predeen piettac-eeitiion transform which can transform noisy data to clean data. Motivated by this property, we initialize the implicit generator go for Diff-Instruce in Algorithm I via score network-induced data-prediction transform of the teacher diffusion models at some fixed time t":
x 2 == s,))))
(B.3)
where 2 <>:N(==; I)) is the latent vector. This generator takes a Gaussian noise with the variance (t") and zero mean an input latent vector.
We find that each noise level (tt)) can give a comparable initialization, so we roughly selected an (t") for different datasets. We put detailed hyper- parameters for distilling in Table 7 The left hand of ggives a demonstration of the samples with the same latent vectors from different generators during training. The top line is the initialized generator (i.e. modified from pre-trained diffusion models). During the training, the initialized generator trained to generate high-quality samples. The right hand of [ class-conditioned samples from one-step model (generator) distilled from a class-conditional EDM model on the CIFAR 10 dataset.
Comparison of Computational Costs The Diff-Instrud algorithm involves training an additional auxiliary diffusion model during the training phases. It is worth noting that although this auxiliary diffusion model brings additional memory cost, this additional memory cost is very limited because the memory bottleneck of training lies in the computational graph of the backpropagation, instead of only saving one more model.
In the Diff-IInstruct algorithm, the model and the generator are updated alternatively, which means that the other model's 's parameters are fixed and do not participate in back-propagatte when one model being updated. So the memory cost for back sock-proaggatin through the computational graph is almost the same as one model.
To quantitatively measure how much additional computational costs brought in, run an experiment to compare the computational and memory costs of Diff-Instruct with our baseline method, the Consistency Distillation, 8 The test was run on 2 Nvidia V100 GPUs with 128 batch size and PyTorch distributed data- pataalllllll mechanism.
The result shows that Diff- Instrutttttt brings in minor additional memory costs than CD (10.40 over 9.55). This is because the Instructt only needs additional GPU memory to save the auxiliary model sg- But the 82 and generator 9o updated alternatively, so their computational graph does not interact. As a result, the memory bottleneck caused by computational graph and back pec-prropgattion does not bring more costs to Diff-Instae Diff-Innttuut.
As for the wall-clock time for IK iterations, we see that Diff-Instrun costs 0.0728 seconds, while the CD costs 0.0489 seconds. This is because each iteration of Diff-Instruct consists of two alternate steps as we show in Algorithm Overall, the Dnsf-Instruct costs almost the same GPU and CPU memory as the bascline CD, but about 1.5 times wall-clock time than the CD for each iteration.
21

---

Table 9: Hyperparameters used for Diff-Instruct for GAN Improvement on CIFARIO dataset under
different settings.
Hyperammmnter
SYRCGAN-22 Cond
StylGGAN Uncond
StyItGAN---AAA Cond
SyIRIAN:-AAJA Uncond
DM
Genemtirr (V
DM
Genentoo
DM K.
Genertior
DM
Generatorg go
Learning rike
22-7
25-7
1,50.6
1.32-6
1.52.
1,55-
1,526
1.5c-
Batch size
512
512
512
512
512.
512
512
S12
Adamm
0.0
0.0
0.0
0.0
0.0
Adaon
0,99
0.99
0199
Figure 5: Generated samples from improved generators with Diff-IIntrun in Table 3 and 1 for generated samples from left to is 2.71, 6.62, and 7.56.
The FID
B.2 Detailed experimental settings of GAN improvement
This experiment aims to show the power Disf-Instruct to improve GAN's generator under different settings transferring knowledge from pre-trained DMs.
Experiment settings. The StyleGAN-2 model is competitive GAN model on benchmarking datasets such as the CIFAR10 10 dataset, so we take StyleGAN-2 the representative GAN model. We take the pre-trained EDM models with VP architecture on the CIFARIO datasets as pre-trained diffusion models.
We download the checkpoint of the pre-trained StyleGAN-2 models with Adaptive Data Augmentation (ADA) (29 from the official website We pre-train StyleGAN-2 model [30] following the same configuration of the original paper. All models converge with adversarial training under different settings (conditional or unconditional with different data augmentation strategies). We initialize generator with the weights of pre-trained GAN generators. We initialize the DM 89 for implicit distribution with the same weights as the pre-trained DMs. We then use the Diff-Instruct with pre-trained DMs to improve the generator. For both the implicit DM & and the generator Gs. we use the Adam optimizer to update the parameters. For the generator, average technique as the official implementation of StyleGAN-ADA with Pytorch. We put detailed hyper-parameters for each experiment in Table
the same exponential moving
22


Reviews:
[{"review_id": "krcl6y6hp7", "reviewer": "Reviewer_AdFD", "summary": "The paper proposes a general framework for distilling a pretrained diffusion model into an arbitrary one-shot latent-variable model. The objective is a reverse KL divergence integrated over time, and an extra score model is learned for the current model distribution. The generator model parameter and the score model parameter are updated alternatively. Empirical results were shown on distilling a diffusion model to a model of same architecture, and finetuning a pretrained GAN model. ", "questions": "- Fig.1 is pretty low-res. Consider improving the figure quality.\n- There's a concurrent work (not a criticism!): ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, which proposed essentially the same training objective in the context of text-to-3D and justified the objective from W gradient flow. Would be nice to compare with and discuss this work. ", "limitations": "Yes.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposed a general framework for distilling pretrained diffusion model on a model-architecture agnostic fashion. Meaning that the distilled model doesn't have to have the same model architecture as many other distillation approaches for diffusion model. This can be useful for many potential applications such distilling 2D diffusion models for 3D.\n- Introducing a learnable score network for the generator model removes the need of learning an inference network as the usual variational inference requires. \n- Overall speaking I think this framework is nice and can potential have impact not only for 2D image distillation / refining, but also cross-modality knowledge transferring. ", "weaknesses": "- One piece missing and I'm eager to see is, if the implicit model is randomly initialized with an arbitrary architecture (e.g. a styleGAN generator), would this approach work? The experiments show that it works well if the implicit model is either initialized from a pretrained DM or a pretrained GAN model. It makes me doubt if this approach works only when the generator is already well-initialized. Adding this piece will make the paper much more stronger IMO.\n-  The approach involves training two models jointly which inevitably introduces the issue of mismatching between two model classes and potentially results in instable training. I'd like to see more analysis towards this issue. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions, we will take them in the revision. In the following paragraphs, we will address your concerns one by one. \n\n**Q1**. If the implicit model is randomly initialized with an arbitrary architecture (e.g. a StyleGAN generator), would this approach work?\n\n**A1**. In our exploration, we have tried and found distilling a random initialized generator **also works** for both UNet and StyleGAN generators, but there takes much more time and is relatively unstable than the one that incorporates knowledge of pre-trained diffusion models. Besides, our goal is to transfer as much knowledge as possible from pre-trained diffusion models, so we choose to initialize the generator with the pre-trained model.  As a simple demonstration, we visualize a batch of class-conditional generated samples from a StyleGAN2 generator that is randomly initialized and trained with Diff-Instruct in **Figure 2** on the additional page in the global author rebuttal cell. Thank you for the suggestion, and we plan to add more discussions on training from scratch in our work in the revision. \n\n**Q2**. The approach involves training two models jointly, ... ... and potentially results in unstable training.\n\n**A2**. Thank you for the reminder. First, we want to say that our two models are trained alternatively, instead of jointly. Second, empirically, we find that the training curve of Diff-Instruct in all experiments is quite stable and fast. To demonstrate the comparison, we give **Figure 2** on the additional page in the global author rebuttal cell to show the FID curve with respect to training iterations of Diff-Instruct. The Diff-Instruct's FID on CIFAR10 distillation converges to 4.6 within 7K iterations. We think the reason for such stable training is that the diffusion models have been developed to be able to quickly match the generator's marginal score functions. Besides, we initialize $s_\\phi$ with the same weights of teacher model $s_{q^{(t)}}$, which we find helps to stabilize the training than a random initialized $s_\\phi$. Since the $s_\\phi$ is initialized the same way as $s_{q^{(t)}}$, the gradient for generator $g_\\theta$ is small (equation 3.2) at the early stage of the distillation, this makes the $s_\\phi$ has sufficient time to match marginal scores of $g_\\theta$ distribution.\n\n**Q3**. Fig.1 is pretty low-res. Consider improving the figure quality.\n\n**A3**. Figure 1 is of high resolution, and we guess you are talking about Figure 2. Figure 2 shows generated samples from ImageNet64 and CIFAR10 of 32 resolutions, so their resolutions are low. \n\n**Q4**. There's a concurrent work (not a criticism!): ProlificDreamer, Would be nice to compare with and discuss this work.\n\n**A4**. Thanks for the reminder. It's interesting that ProlificDreamer proposes essentially the same objective as Diff-Instruct in the concurrent work, but we do not know this work before our submission. We plan to add the discussion to this work in our revision. Our paper and ProlificDreamer focus on different aspects. We use Diff-Instruct to obtain state-of-the-art single-step diffusion distillation performance and outperform adversarial training for GAN generators. Instead, the ProlificDreamer specializes in the scenario of text-to-3D creation and shows a really impressive performance. As a brief conclusion, we think both Diff-Instruct and the ProlificDreamer contribute to the research direction that aims to better use the knowledge of pre-trained diffusion models. \n\nThank you for your kind suggestions. We hope our response has resolved your concerns. If you still have any concerns, please let us know."}, {"review_id": "h7TByvULtA", "reviewer": "Reviewer_Ln4e", "summary": "The paper proposes a method for data free knowledge distillation from large scale diffusion models. It tries to distill the knowledge contained into a diffusion model, into another diffusion model or an explicit image generation model like a GAN. They propose a novel divergence measure obtained by integrating the KL divergence along the path of the diffusion model to match distilled model with the original instructor model. They show that the gradient update for the generator can simply be obtained as a function of the score functions as long as the generator is a differentiable function in its output. Using this the paper shows that they are able to distill knowledge from pre-trained diffusion models into generative models like GANs which enables faster inference. The paper provides empirical results to support their claims.", "questions": "1. How does the generator converge to the distribution of the pre-trained diffusion model?\n2. Can you reduce the number of reverse diffusion step in the first step of algorithm 1 so that its inference is comparable to the generator? Since the generator indirectly interacts with the pre-trained diffusion model.\n3. How will simple score matching work, with lesser number of diffusion steps? What’s the need to introduce a generator model? Is it some form of regularization?", "limitations": "1. Authors need to provide an elaborate discussion comparing the proposed method with its baselines for instance consistency models, and justify why the method is better compared to the baseline in terms of the theoretical convergence or inference", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper discusses an important problem of distilling pre-trained diffusion models, especially into single-shot prediction models which enables faster inference.\n2. The proposed method is simple to understand and implement in practice.\n3. Since the generator is a single-step predictor, it provides extremely fast inference at test time compared to standard diffusion models.", "weaknesses": "1. The convergence of the proposed algorithm is not clear in the paper. The authors should elaborate as to why will the generator generate images which resemble the pre-trained diffusion model. In the absence of the generator, one can always sample random images and try to match a new diffusion model to a pre-trained model. Why will this approach not work? (One can use lesser number of reverse diffusion steps for this model)\n2. This method requires training two models, which increases the space complexity of training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews. We will address your concerns one by one in the following paragraphs. Before that, we first give a summary of the main contributions of our work. \n\nIn this work, we introduce the Diff-Instruct, which can utilize a pre-trained teacher diffusion model to train a student one-step generator. To do this, we minimize the Integral KL divergence between the generator's and the teacher diffusion model's distribution. We use an auxiliary score network that is trained to approximate the generator's marginal score functions and used to provide updating gradient for the generator. After training, the auxiliary score net is discarded, and the learned student's one-step generator is used for fast generation. \n\nNext, we address your concerns one by one. \n\n**A1**. (1) We are not sure what you mean by \"one can always sample random images and try to match a new diffusion model\". We guess you are assuming that the one-step generator is used to help obtain a fewer-step diffusion model, which is similar to what Progressive distillation (PD) does. However, the goal of the Diff-Instruct is to train the one-step generator, instead of a fewer-step diffusion model. So we do not need to sample from diffusion models. To train the one-step generator, we aim to minimize the IKL divergence through Algorithm 1. In algorithm 1, we alternate between training an auxiliary score network $s_\\phi$ to learn the generator's marginal score functions and using $s_\\phi$ with teacher model $s_{q^{(t)}}$ to provide a gradient for the generator's parameters to minimize the IKL divergence. The algorithm is stable and shows strong empirical performance. If the generator and score network has unlimited expressive ability, then the generator's distribution will converge to the teacher diffusion model's underlying distribution. \n\n(2) Since the one-step generator is what we want and we do not need the diffusion model to sample, we think the concerns in questions, quotes \n<1>\"In the absence of the generator\",\n<2>\"use a lesser number of reverse diffusion steps\",\n<3>\"What’s the need to introduce a generator model?\",\n<4>\"Can you reduce the number of reverse diffusion step in the first step of algorithm 1\"\nhave no relation with Diff-Instruct. I am not sure whether I have addressed your concerns. Please let me know if they are not your concern.\n\n**A2-A5**. We acknowledge that the additional model brings additional memory cost for saving additional models. However, this additional memory cost is limited because the memory bottleneck of training lies in the computational graph of the backpropagation, instead of only saving one more model. In Diff-Instruct, the model $s_\\phi$ and the generator $g_\\theta$ are updated alternatively, which means that the other model's parameters are fixed and do not participate in back-propagation when one model is being updated. So the memory cost for back-propagating through the computational graph is almost the same as one model. To quantitatively measure how much additional computational costs are brought in, we compare Diff-Instruct and CD's memory and computational costs in **Table 3** in the global author's rebuttal cell. \n\nThe result show Diff-Instruct costs almost the same GPU and CPU memory as the baseline CD, but about 1.5 times wall-clock time than the CD for each iteration. \n\nBesides, our empirical evaluation also reveals that the Diff-Instruct converges significantly faster than the baseline method, the consistency distillation (CD). We record the FID curve w.r.t training iterations of Diff-Instruct and CD in the Left part of **Figure 2** on the additional pdf page in the global rebuttal cell. The results show that Diff-Instruct converges significantly faster than CD. For instance, the FID of Diff-Instruct converges to 4.6 with 7k training iterations, while the consistency distilled student model still holds an FID over 30. \n\n**A6** (limitations). Thank you for your reminders. There are mainly four differences (and advantages) between Diff-Instruct and baseline CD. \n\n(1) Diff-Instruct trains the student generator by minimizing the IKL divergence. Instead, the CD trains the student model by minimizing the self-consistency metric. To minimize the self-consistency metric, the CD needs an additional exponential moving average (EMA) model which is slowly updated. Also in practice, this EMA model is hard to tuning to achieve the optimal performance. \n\n(2) The Diff-Instruct allows for a flexible student generator whose input and output dimensions are not necessarily the same. This makes it applicable to transferring knowledge from pre-trained diffusion models to various generators in different applications including diffusion distillation, GAN improvement, and text-to-3D creation. On the contrary, CD requires the student model to have the same input and output dimensions, so CD is not applicable for applications such as GAN improvement and text-to-3D generation. \n\n(3) Diff-Instruct shows better empirical performance (the FID) than CD on ImageNet64 one-step distillation Benchmarks. Besides, its performance does not depend on other learned metrics, such as the LPIPS that is used in CD. \n\n(4) Diff-Instruct is data-free, while CD requires ground truth data. This advantage makes the Diff-Instruct have a wider use than CD, especially for cases when real data is difficult or expensive to obtain.\n\nDespite the advantages, Diff-Instruct has its limitations compared with CD. The CD-distilled model is able to run multiple times to obtain better sample quality, while the Diff-Instruct can only distill to a one-step generator. We think the novel extension of Diff-Instruct to support multiple-step inference is an important future direction. \n\nWe hope our answers have resolved your concerns, and if you still have any concerns, please do let us know."}, {"review_id": "o34XXNiXIe", "reviewer": "Reviewer_62Pb", "summary": "The paper introduces a method to transfer the knowledge of a pre-trained diffusion model to an implicit generator model, such that the distribution of the generated samples from the generator matches that of the pre-trained DM. The training is data free, and it can be divided into two alternating phases: training a diffusion model on generated samples from the generator model, and updating the generator model to produce samples that produces similar score with the pre-trained DM. ", "questions": "See above.", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. The paper tackles a very interesting idea. While there are many papers targeting at improve the sampling speed of diffusion models, and multiple works have designed methods to distill diffusion models, the idea of distilling the knowledge to another generator model is, to the best of my knowledge, novel.\n\n2. The method itself is solid and interesting. With the definition of IKL, the training objective becomes an explicit divergence minimization.\n\n3. Results are comprehensive and strong. It shows results on both distilling diffusion models to a single step and enhancing existing GAN models. It also shows impressive performance on single step generation. The fact that it can further improve SOTA GANs is impressive.\n\n", "weaknesses": "1. The presentation of the algorithm can be improved. Figure 1 isn't very informative, and a more detailed introduction should be included in the caption. Algorithm 1 isn't very clear, as it only shows the gradient formulation, without introducing how to compute it. For example, I am not very clear about how to estimate the integral in the gradient formulation. Did you just sample one random t as a very rough approximation? \n\n2. As you mentioned, when we assume the generator output has dirac delta distribution, the formulation is essentially the same as SDS in dreamfusion. But we know that GAN is indeed a generator model with dirac delta distribution, so what is the real benefit of assuming non-dirac delta distribution for the generator? Did you actually find any benefit of training with algorithm 1 with an auxiliary DM over just training with 3.3?\n\n3. I do not quite understand the distillation part. When doing the distillation (instead of GAN enhancement), at early stage the generator just output non-sense images, and matching the score on it can be meaningless. Is there any explanation on how it can be applied to distilling DM to a non-pretrained generator?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable suggestions, we will incorporate them in the revision. In the following paragraphs, we will address your concerns one by one. \n\n**Q1**. The presentation of the algorithm can be improved. I am not very clear about how to estimate the integral in the gradient formulation\n\n**A1**. Thanks. We will revise the representation of Algorithm 1 in the revision. As for the implementation of the gradient estimation, we sample a batch of different times (diffusion sigma levels) and apply forward diffusion on generated images to corresponding times and calculate the gradient with the 2nd step of Algorithm 1. Then we average the gradient over the batch to update the generator. This batch-average manner is the same as training diffusion models. \n\n**Q2**. We know that GAN is indeed a generator model with dirac delta distribution, so what is the real benefit of assuming non-dirac delta distribution for the generator?\n\n**A2**. For a GAN generator, the output distribution is not a single-point distribution, because the input latent vector $z\\sim p_Z$ has randomness. In practice, we tried to use equation (3.3) for distillation, but the trained generator shows strong model-collapse behavior, i.e. the generator collapses on certain modes of data, which leads to a distillation failure. The reason behind the failure of using 3.3 other than 3.2 to update the generator lies in that the general generator's output distribution is not single-point, thus using 3.3 is not suitable. \n\n**Q3**. I do not quite understand the distillation part. When doing the distillation (instead of GAN enhancement), at early stage the generator just output non-sense images, and matching the score on it can be meaningless. Is there any explanation on how it can be applied to distilling DM to a non-pretrained generator?\n\n**A3**. We find that by initializing the generator with the teacher diffusion model, the generator's output has a blurred shape of the generated objects and the correct output range (i.e. pixel values in [-1,1]), this makes the learning of score net and updating generator much easier compared to a purely random initialized generator. \n\nBesides, we find that training random-initialized generators also works. But the training takes significantly more time and is relatively unstable when compared with the well-initialized generator. So we choose to initialize the model with pre-trained diffusion. As a simple demonstration, we visualize a batch of class-conditional generated samples from a StyleGAN2 generator that is randomly initialized and trained with Diff-Instruct in **Figure 2** on the additional page in the global author rebuttal cell. \n\nThank you for your kind suggestions. We hope our answers have resolved your concerns, and if you still have any concerns, please do let us know."}, {"review_id": "pguq4ecUvO", "reviewer": "Reviewer_6Ze7", "summary": "This work distills diffusion models to a GAN-style generator. The author provides mathmatic connection to other works, as well as experimental comparisons between the distilled GAN-style generator with other (mostly) diffusion models. \n ", "questions": "Please check the weaknesses. ", "limitations": "this work has no negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work introduce a way to distill diffusion model to GAN-style generator. \n- And shows the connection to other works.\n- The experiments show the proposed method works well. \n", "weaknesses": "- Lack baseline method. Table1 and Table2 shows the performance many diffusion methods, but they are either diffusion model trained on different datasets or distilled diffusion model that may not use the same teather model as the author used (i.e. EDM). In this case, the numbers in table1 &  2 are not compariable. There should be at least one baseline method that use the same teather model (EDM), the same generator architecture (one-step GAN-style generator), but different distillation method. Here is a simple baseline idea,  generating N images by the teather model (EDM), training a GAN on the generated images. \n\n- Lack time comparison. One big benefit of one-step generator (as well as distillation) is inference speed. Otherwise, we can use the teacher model directly.  \n\n\n\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your reviews.  We will address your concerns one by one in the following paragraphs. \n\n**Q1**. Lack of baseline method.\n\n**A1**. We are sorry for the confusion. In Table 1 and Table 2 in the main text, the CD, PD, and Diff-Instruct use the same teacher model and the same student model. To make our representation more clear, we re-organize Table 1 and Table 2 to **Table1** and **Table2** in order to emphasize each teacher and student model. \n\n**Table1**. Unconditional sample quality on CIFAR10 through diffusion generations. ∗Methods that require synthetic data construction for distillation. †Methods that require real data for distillation.\n| Method                    | Teacher   | Student   | NFE       | FID       | IS        |\n| :------:                  | :------:  | :------:  | :------:  | :------:  | :------:  |\n| Pre-train + distillation  |\n| 1-ReFlow (+distill)∗      | ReFlow    | UNet      | 1         | 6.18      | 9.08      |\n| 2-ReFlow (+distill)∗      | ReFlow    | UNet      | 1         | 4.85      | 9.01      |\n| 3-ReFlow (+distill)∗      | ReFlow    | UNet      | 1         | 5.21      | 8.79      |\n| KD∗                       | DDPM      | UNet      | 1         | 9.36      |           |\n| CD-LPIPS†                 | EDM       | UNet      | 2         | 2.93      | 9.75      |\n| PD                        | EDM       | UNet      | 2         | 5.58      | 9.05      |\n| PD                        | EDM       | UNet      | 1         | 8.34      | 8.69      |\n| CD-L2†                    | EDM       | UNet      | 1         | 7.90      | 8.69      |\n| CD-LPIPS†                 | EDM       | UNet      | 1         | 3.55      | 9.48      |\n| Diff-Instruct             | EDM       | UNet      | 1         | 4.53      | 9.89      |\n\n**Table2**. ImageNet 64×64 through diffusion generations. ∗Methods that require synthetic data construction for distillation. †Methods that require real data for distillation.\n| Method                    | Teacher   | Student   | NFE       | FID       |\n| :------:                  | :------:  | :------:  | :------:  | :------:  |    \n| Pre-train + distillation  |\n| GGDM∗                     | DDPM      | UNet      | 25        | 18.4      | \n| PD                        | EDM       | UNet      | 2         | 8.95      | \n| CD-LPIPS†                 | EDM       | UNet      | 2         | 4.70      | \n| PD                        | EDM       | UNet      | 1         | 15.39     | \n| CD-L2†                    | EDM       | UNet      | 1         | 12.10     |  \n| CD-LPIPS†                 | EDM       | UNet      | 1         | 6.20      | \n| Diff-Instruct             | EDM       | UNet      | 1         | 5.57      | \n\nIn particular, Table 1 and Table 2 in the main text show that the EDM model is the strongest among all diffusion models, so in our work, we use EDM models as the teacher models. In **Table 1** and **Table 2**, PD, CD, and Diff-Instruct all use the EDM teacher, and the same UNet student with the same architecture as the teacher model. (The author of consistency distillation re-implemented the PD for EDM, so the reported FID for PD is lower than that in PD's original paper)\n\nFor the GAN improvement experiment, we compare the StyleGAN2 generator trained with and without Diff-Instruct in Tables 3 and 4. The StyleGAN2 generator is obtained from the official release. We use the pre-trained EDM as the teacher model for Diff-Instruct. \n\n**Q2**. Lack time comparison. One big benefit of one-step generator (as well as distillation) is inference speed. Otherwise, we can use the teacher model directly.\n\n**A2**. In Table 1 in the main text, the Diff-Instruct generator uses the same UNet architecture as the teacher diffusion model, so the number of sampling steps (NFE) represents its inference time costs. As we show in the upper part of Table 2 in the main text, sampling from our learned one-step generator with 1 NFE results in an FID of 4.19, which is significantly better than its teachers with 10 NFEs with an FID of 15.56. So we conclude that the one-step generator trained with Diff-Instruct achieves at least 10+ times acceleration (more efficient) than its teacher diffusion model.  \n\nThank you for your kind suggestions. We hope our answers have resolved your concerns, and if you still have any concerns, please let us know. "}, {"review_id": "FY24z5OaFu", "reviewer": "Reviewer_kX6X", "summary": "The paper discusses the distillation of pre-trained diffusion models. The paper introduces a framework called Diff-Instruct, which allows for the transfer of knowledge from pre-trained DMs to other generative models in a data-free manner. Diff-Instruct utilizes a mathematical foundation based on minimizing a divergence called Integral Kullback-Leibler (IKL) divergence. This divergence is specifically designed for DMs and is more robust in comparing distributions with misaligned supports. The text also highlights connections to existing works such as DreamFusion and generative adversarial training. The effectiveness of Diff-Instruct is demonstrated through two scenarios: distilling pre-trained diffusion models and refining existing GAN models. Experimental results show that Diff-Instruct achieves state-of-the-art performance in single-step diffusion-based models and consistently improves pre-trained generators of GAN models across various settings.", "questions": "* When discussing knowledge transfer, what is the primary objective? Is it knowledge distillation or transfer? It would be helpful if you could simplify and specifically tie your contributions to the results of your experiments.\n\n* Regarding Algorithm 1: when updating $\\theta$, shouldn't $s$ be function of $\\theta$? Could you clarify how the gradient has been formulated?\n\n* In Equation 1, does $q$ represent the forward or reverse process in a Diffusion Model? It appears you're using $q$ for the reverse process of a pre-trained Diffusion Model, which is conventionally represented as the approximate posterior. Could you clarify this point?\n\n* Furthermore, referencing Equation 1 in this paper (https://arxiv.org/abs/2011.13456), it seems that the notations in Equations 2.2 and 3.2 are inconsistent. Could you please provide some clarification?\n\n* After training the implicit model, what role does $s_{\\phi}$ serve? Is it merely an auxiliary model, and if so, how do you initialize it?\n\n* How much time is required to distill a pre-trained diffusion model?\n\n* You've mentioned that the Inverse Kullback-Leibler (IKL) divergence is more robust. Could you explain how you've arrived at this conclusion?", "limitations": "* Notation confusing or inconsistent.\n\n* Not clear what is the goal of the paper (transfer vs distillation).\n", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* The paper tackles the problem of distilling diffusion models and addresses the important topic of accelerating inference for DMs. \n\n* The proposed method utilizes a novel formulation based on the Integral Kullback-Leibler (IKL) divergence, which adds further interest to the approach. The authors make an effort trying to explain why this divergence is a reasonable choice for distillation.\n\n* The experimental results showcased the effectiveness of this method, particularly in the context of 1-step distillation on datasets such as CIFAR and ImageNet64, where the results were highly promising. \n\nThese findings demonstrate the potential of the proposed method for enhancing the efficiency and performance of diffusion models in small domains.", "weaknesses": "* While the distillation results for CIFAR and ImageNet using the proposed method are good, they may not have practical utility in larger domains. Distillation becomes particularly crucial for such larger domains, and it remains unclear whether the method will be effective with larger images. \n\n* Additionally, although the method doesn't require additional data, it does necessitate the use of two extra models, namely the student model and an auxiliary model for gradient propagation. This requirement can be prohibitive when dealing with larger domains and bigger models. \n\n* I have some doubts about the notation (see questions). I would try to be consistent with the notation in the literature (see https://arxiv.org/abs/2011.13456).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful feedback. We will address your concerns one by one in the following paragraphs. \n**Q1**. it remains unclear whether the method will be effective with larger images\n\n**A1**. We agree that larger-scale experiments, such as the text-to-image generation experiments are important to show the scaling ability of Diff-Instruct. However, due to the massive computational requirements and engineering implementations for large-scale generative modeling, we leave the scaling of Diff-Instruct to larger domains in future work.  However, in the rebuttal period, we try the Diff-Instruct on distilling Stable Diffusion into one step and got positive results. We put a visualization of a batch of images that are generated by a one-step generator distilled from Stable Diffusion in **Figure 1** on the additional page in the global author rebuttal cell. Our results indicate the scalability of Diff-Instruct to large-scale text-to-image diffusion models, and we plan to leave more exploration and engineering in our future work. \n\n**Q2**. two extra models and costs\n\n**A2**. We acknowledge that the additional diffusion model ($s_\\phi$ in Algorithm 1) brings additional memory cost. However, this additional memory cost is limited because the memory bottleneck of training lies in the computational graph of the backpropagation, instead of only saving one more model. In Diff-Instruct, the model $s_\\phi$ and the generator $g_\\theta$ are updated alternatively, which means that the other model's parameters are fixed and do not participate in back-propagation when one model is being updated. So the memory cost for back-propagating through the computational graph is almost the same as one model. To quantitatively measure how much additional computational costs are brought in, we compare Diff-Instruct and CD's memory and computational costs in **Table 3** in the global author's rebuttal cell. \n\nThe result show Diff-Instruct costs almost the same GPU and CPU memory as the baseline CD, but about 1.5 times wall-clock time than the CD for each iteration. \n\nBesides, our empirical evaluation also reveals that the Diff-Instruct converges significantly faster than the baseline method, the consistency distillation (CD). We record the FID curve w.r.t training iterations of Diff-Instruct and CD in the Left part of **Figure 2** on the additional page in the global author rebuttal cell. The results show that Diff-Instruct converges significantly faster than CD. For instance, the FID of Diff-Instruct converges to 4.6 with 7k training iterations, while the consistency distilled student model still holds an FID over 30. \n\n**Q4**. When discussing knowledge transfer, what is the primary objective?\n\n**A4**. We are sorry for the confusion of the word transferring and distillation. Since these two words do not have a clear definition in the literature, we abuse their use to mean to use a pre-trained diffusion model to guide the training of a one-step generator. \n\n**Q5**. Could you clarify how the gradient has been formulated?\n\n**A5**. In algorithm 1, when updating $\\theta$, we fix the parameter of $s_\\phi$ and use $s_\\phi(.,t)$ and pre-trained score $s_{q^{(t)}}$ to compute the gradient of the parameter $\\theta$. In the gradient formula, for a fixed time $t$, the term $x_0 = g_\\theta(z)$ and $x_t|x_0\\sim q_t(x_t|x_0)$ contain parameter $\\theta$. Other components do not contain $\\theta$. As for the implementation of the gradient, we random sample $x_0 = g_\\theta(z)$ and $x_t|x_0\\sim q_t(x_t|x_0)$, then we compute a coefficient of $coeff = w(t)[s_\\phi(x_t) - s_{q^{(t)}}(x_t)].\\operatorname{detach()}$. The $\\operatorname{detach()}$ operator means to cut off the parameter dependency of the current node in the computation graph. Then we compute a loss function $\\operatorname{loss}(\\theta) = \\langle coeff, x_t(\\theta) \\rangle$ and backward through the loss functions. So we have $\\frac{\\partial}{\\partial\\theta}\\operatorname{L}(\\theta) = w(t)[s_\\phi(x_t) - s_{q^{(t)}}(x_t)]\\frac{\\partial x_t(\\theta)}{\\partial}$, which implements the gradient formula of Diff-Instruct at a certain time level. \n\n**Q3,6-7**. some doubts about the notation. does represent the forward or reverse process in a Diffusion Model?\n\n**A3,6-7**. In equations (2.2), (3.2), and Algorithm 1, We use the notion $q_t(x_t|x_0)$ to represent the forward process and the notion $q^{(t)}$ for the forward marginal distribution. This is different from Eq. (1) of the score sde paper which uses $p$ to represent the forward. \n\n**Q8**. After training the implicit model, what role does  serve? how do you initialize it?\n\n**A8**. After training, we discard the auxiliary model $s_\\phi$ and only take the generator $g_\\theta$ for inference. For all experiments, we initialize $s_\\phi$ using the same architecture and weights of the pre-trained model $s_{q^{(t)}}$. This initialization works well across all experiments. \n\n**Q9**. How much time is required to distill a pre-trained diffusion model?\n\n**A9**. In practice, we find that using Diff-Instruct to train a one-step generator on CIFAR10 to converge takes less than 8 wall-clock hours on a single V100 GPU (with batch size 96), this shows that Diff-Instruct is quite efficient. \n\n**Q10**. Explain why IKL is more robust. \n\n**A10**. In Appendix A.1, we show that under a famous example with misaligned support from the Wasserstein GAN paper, the KL divergence is ill-defined (diverges to infinity)  due to misaligned density support. While the IKL is well-defined because it integrates the divergence after diffusion. This shows that IKL is more robust than KL divergence when measuring the distributions that have misaligned supports.\n\nThank you for your kind suggestions. We hope our answers have resolved your concerns, and if you still have any concerns, please do let us know."}]

Author rebuttal:
Thank all reviewers for your valuable feedback. In the rebuttal period, we run an additional experiment to compare the computational and memory costs of Diff-Instruct with our baseline method Consistency Distillation in Table 3. The test was run on 2 Nvidia V100 GPUs with 128 batch size and PyTorch distributed data-parallel mechanism. 

**Table3**. Comparison of Memory Costs and Wall-Clock Timing of CD and Diff-Instruct. The Peak GPU-Memo shows the Maximum observed GPU memory caused by the Tensor and Computational Graph of the host GPU. Sec-per-K Iterations report the wall-clock time for each training iteration. For Diff-Instruct, each iteration consists of two stages as in Algorithm 1 in the main text. Test environment: PyTorch 1.12.1 and Torchvision 0.13.1, and Torch.distributed.parallel on 2 V100 GPUs.
| Method        | Peak GPU-Memo(GB) | Peak CPU-Memo(GB) | Sec-per-K Iterations  |
| :------:      | :------:          | :------:          | :------:              |
| CD            | 9.55              | 2.75              | 0.0489                |
| Diff-Instruct | 10.40             | 2.78              | 0.0728                |

The result shows that Diff-Instruct brings in minor additional memory costs than CD (10.40 over 9.55). This is because the Diff-Instruct only needs additional GPU memory to save the auxiliary model $s_\phi$. But the $s_\phi$ and generator $g_\theta$ are updated alternatively, so their computational graph does not interacts. As a result, the memory bottleneck caused by computational graph and back-propagation does not bring more costs to Diff-Instruct.

As for the wall-clock time for 1K iterations, we see Diff-Instruct costs 0.0728 seconds, while the CD costs 0.0489 seconds. This is because each iteration of Diff-Instruct consists of two alternate steps as we show in Algorithm 1. Overall, the Diff-Instruct costs almost the same GPU and CPU memory as the baseline CD, but about 1.5 times wall-clock time than the CD for each iteration. 

Besides the computational costs, we also show the convergence of FID of Diff-Instruct and CD in the left part of **Figure 2** on our additional page of figures. In that Figure, the Diff-Instruct's FID on CIFAR10 distillation converges to 4.6 within 7K iterations, while the CD distilled model's FID still remains larger than 30. This shows the Diff-Instruct converges quickly with iterations. 

Meta-review: This paper has garnered mostly positive reviews, while the main concerns raised in the reviews (having mostly to do with the convergence of the algorithm and the relation to few-step diffusion models) seem to have been properly addressed in the rebuttals.
Reviewers appreciated the clear motivation, the formal foundation of the proposed method and the clarity of its implementation.
Reviewers were also positively impressed expressed by the comprehensive and convincing experimental results validating the method.
Finally, reviewers concluded that the impact of this work might even reach beyond distilling models for image generation, and extend to cross-modality applications.
Decision: Accept (poster)