SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The fact that SDS is a special case of Diff-Instruct is not a coincidence, as in DreamFusion, the rendered image of a NeRF model from a certain camera view is a 2D image that is differentiable to NeRF's parameters. Therefore, using SDS to learn a NeRF model is essentially an application of using Diff-Instruct to distill a pre-trained text-to-2D diffusion model in order to obtain a 3D NeRF object. However, the path we obtain (3.3) is totally different from that in DreamFusion. In DreamFusion, the authors obtained (3.3) by taking the data gradient of the diffusion model's loss function (2.2) and empirically omitted the Jacobian term of the pre-trained score network. However, in this work, we first propose the general formulation of Diff-Instruct and then specialize it to obtain SDS in a natural way.
3.3.2 Connection to GANs
Our proposed Diff-Instruct without integral on time is equivalent to the adversarial training [19] that aims to minimize the KL divergence. Following the same notation as in Section 2, the adversarial training uses a discriminator  $h(. )$  to learn the density ratio to construct the objective for the generator.
Corollary 3.5. If the discriminator  $h$  learns the perfect density ratio, i.e.  $h(\pmb{x}) = \frac{p_d(\pmb{x})}{p_d(\pmb{x}) + p_g(\pmb{x})}$ , then updating the generator to minimize the KL divergence  $(\mathcal{L}_g^{KL})$  in Section 2 is equivalent to Diff-Instruct with a weighting function  $w(0) = 1$  and  $w(t) = 0, \forall t > 0$ .
The Diff-Instruct is essentially a different method from adversarial training in three aspects. First, the adversarial training relies on a discriminator network to learn the density ratio between the model distribution and data distribution. However, Diff-Instruct employs DMs instead of discriminators to instruct the generator updates. Second, in scenarios where only a pre-trained diffusion model is available without any real data samples, Diff-Instruct can distill knowledge from the pre-trained model to the implicit generative model, which is not achievable with adversarial training. Third, Diff-Instruct uses the IKL as the minimization divergence, which overcomes the degeneration problem of the KL divergence via a novel use of diffusion processes and can potentially overcome the drawbacks such as mode-drop issues of adversarial training.
3.4 Related works
The pre-trained diffusion models on large-scale datasets contain rich knowledge of the data distribution. There is a growing interest in distilling this knowledge to other models [43] that are more sampling-efficient, such as implicit generators [50, 26] and neural radiance fields models [48, 54].
One significant advantage of our Diff-Instruct framework is its ability to update the generator without real data  $\boldsymbol{x} \sim p_d$ . Instead, the knowledge of the data distribution to update the generator is contained
3We switch the notation from  $g_{\theta}(z)$  to  $g(\theta)$  since under the assumptions the generator has no randomness.
in the marginal score function  $s_{p^{(t)}}$  as in (3.2). This enables our Diff-Instruct to distill knowledge from pre-trained DMs into flexible generators in a data-free manner and provides a major advantage over other diffusion distillation methods that require either the real data or synthetic data from pre-trained DMs. More precisely, There are three kinds of diffusion distillation methods depending on how to use the data for distillation. The first is the data synthetic distillation, which requires using the pre-trained DM to synthesize data from random noises. The student model then learns the mechanism between random noise and synthetic data in order to enhance the efficiency of data generation. Representative methods of data-synthetic distillation are Knowledge Distillation (KD [42]), Rectified Flow (ReFlow [40]) and DFNO ([82]). The second method does not require synthetic data from diffusion models but involves real data when distilling. The consistency distillation (CD [66]) is a representative method. The third method is pure data-free distillation, which requires neither real data nor synthetic data. The Diff-Instruct and the Propressive Distillation (PD [61]) are representative pure data-free distillation. Generally, we use the word "data-free distillation" to include both data-synthetic distillation and pure data-free distillation. To give a comprehensive comparison of diffusion distillation methods, we summarize three main features of diffusion distillation methods in Table 6. The efficiency represents the computational cost of the distillation method. Methods that require synthesizing dataset is inefficient. The data-synthetic distillation requires simulations with pre-trained DMs, thus is inefficient. The flexibility represents whether the distillation approach is capable of distilling knowledge of pre-trained DM to flexible generator architectures, for example, the generator whose input and output dimension is different. Diff-Instruct is the only method that can apply to a wide variety of downstream generators.
Furthermore, Diff-Instruct offers very high flexibility to the generator, distinguishing it from traditional diffusion distillation methods that impose strict constraints on the generator selection. For instance, the generator can be a convolutional neural network (CNN)-based or a Transformer-based image generator such as StyleGAN [28, 30, 31, 37], or an UNet-based generator [73] adapted from pre-trained diffusion models [32, 65, 22]. The versatility of Diff-Instruct allows it to be adapted to different types of generators, expanding its applicability across a wide range of generative modeling tasks. In the experiment sections, we show that Diff-Instruct is capable of transferring knowledge to generator architectures including both UNet-based and GAN generators respectively. To the best of our knowledge, the Diff-Instruct is the first approach to efficiently enable such a data-free knowledge transfer from diffusion models to generic implicit generators.
With the abundance of powerful pre-trained DMs with diverse expertise, our proposed Diff-Instruct unlocks their potential as sources of knowledge to instruct a wide variety of models. To demonstrate, we choose the state-of-the-art DMs from the seminal work by [32] (we denote as EDMs) as instructors and consider transferring their knowledge to implicit generators. In this section, we evaluate the efficacy of DI through two downstream applications: diffusion distillation and improvement of GAN's generator. These two experiments correspond to using UNet and GAN's generator to absorb the knowledge from DMs. In the diffusion distillation experiments, we use Diff-Instruct to distill pre-trained DMs to single-step generative models. On the ImageNet  $64 \times 64$  dataset, our Diff-Instruct achieves state-of-the-art performance in terms of FID among all single-step diffusion-based generative models. Furthermore, in the GAN-improving experiments, we use Diff-Instruct to improve existing GAN models that are pre-trained to convergence with adversarial training. The Diff-Instruct is shown to be able to consistently enhance the generative performance of pre-trained StyleGAN-2 models on the CIFAR10 dataset by incorporating knowledge of pre-trained DMs.
4.1 Single-step diffusion distillation
Diffusion distillation is a hot research area that aims to accelerate the generation speed of diffusion models. In our experiments, we utilize our Diff-Instruct framework to train single-step generators on CIFAR-10 [36] and ImageNet  $64 \times 64$  [11] from pre-trained EDM [32] models. We evaluate the performance of the trained generator via Frechet Inception Distance (FID) [21], the lower the better, and Inception Score (IS) [62]), the higher the better. For additional details about the generator's architecture, pre-trained models, and the hyper-parameters on our experimental setup, please refer to Appendix B.1.
Performances. Table 1 and 2 summarize the FID and IS of the single-step generator that we trained with Diff-Instruct from pre-trained EDMs on the CIFAR10 datasets (unconditional without labels)
Figure 2: Generated samples from one-step generators that are distilled from pre-trained diffusion models on different datasets. Left: FFHQ-64 (unconditional); Mid: ImageNet-64 (conditional); Right: CIFAR-10 (unconditional).
Table 1: Unconditional sample quality on CIFAR-10 through diffusion generations. *Methods that require synthetic data construction for distillation. †Methods that require real data for distillation.
METHOD	NFE (↓)	FID (↓)	IS (↑)
Multiple Steps (include Diffusion Distillation)
DDPM [22]	1000	3.17	9.46
LSGM [68]	147	2.10
PFGM [74]	110	2.35	9.68
EDM [32]	35	1.97
DDIM [64]	50	4.67
DDIM [64]	10	8.23
DPM-solver-2 [41]	12	5.28
DPM-solver-3 [41]	12	6.03
3-DEIS [78]	10	4.17
UniPC [80]	8	5.10
UniPC [80]	5	23.22
Denoise Diffusion GAN(T=2) [73]	2	4.08	9.80
PD [61]	2	5.58	9.05
CT [66]	2	5.83	8.85
CD† [66]	2	2.93	9.75
Single Step
Denoise Diffusion GAN(T=1) [73]	1	14.6	8.93
KD* [42]	1	9.36
TDPM [83]	1	8.91	8.65
1-ReFlow [40]	1	378	1.13
CT [66]	1	8.70	8.49
1-ReFlow (+distill)* [40]	1	6.18	9.08
2-ReFlow (+distill)* [40]	1	4.85	9.01
3-ReFlow (+distill)* [40]	1	5.21	8.79
PD [61]	1	8.34	8.69
CD-L2† [66]	1	7.90
CD-LPIPS† [66]	1	3.55	9.48
Diff-Instruct	1	4.53	9.89
Table 2: Class-conditional sample quality on CIFAR-10 and ImageNet  $64 \times 64$  through diffusion generations. *Methods that require synthetic data construction for distillation. †Methods that require real data for distillation.
METHOD	NFE (↓)	FID (↓)
Multiple Steps (include Diffusion Distillation)
EDM [32]	35	1.79
EDM-Heun [32]	20	2.54
EDM-Euler [32]	20	6.23
EDM-Heun [32]	10	15.56
Single Step
EDM [32]	1	314.81
Diff-Instruct	1	4.19
Class-conditional ImageNet 64 × 64,†Distillation techniques.
METHOD	NFE (↓)	FID (↓)
Multiple Steps
ADM [12]	250	2.07
SN-DDIM [4]	100	17.53
EDM [32]	79	2.44
EDM-Heun[32]	10	17.25
GGDM [70]	25	18.4
CT [66]	2	11.1
PD†[61]	2	8.95
CD†[66]	2	4.70
Single Steps
EDM[32]	1	154.78
PD†[61]	1	15.39
CT [66]	1	13.00
CD-L2†[66]	1	12.10
CD-LPIPS†[66]	1	6.20
Diff-Instruct	1	5.57
and the conditional generation on the ImageNet  $64 \times 64$  data. Diff-Instruct performs competitively across all datasets among single-step and multiple-step diffusion-based generative models, which involve both models from diffusion distillation or direct training.
As shown in Table 2, on the ImageNet dataset of the resolution of  $64 \times 64$ , Diff-Instruct outperforms diffusion-based single-step generative models in terms of FID, including both distillation methods that require real data or synthetic data, and even models that are trained from scratch. On the unconditional generation of the CIFAR10 dataset, Diff-Instruct achieves the state-of-the-art IS among diffusion-based single-step generative models but achieves the second-best FID, only worse than the consistency distillation (CD) [66] which requires both real data for distillation and the learned neural image metric (e.g. LPIPS [79]). The conditional generation experiment on the CIFAR10 dataset shows that the Diff-Instruct performs better than a 20-NFE diffusion sampling from EDM model [32] with Euler-Maruyama discretization but worse than a 20-NEF Heun discretization.
Figure 3: Comparison of FID convergence on diffusion distillation of CIFAR10 unconditional EDM model [32]. The FID value of consistency distillation is obtained from its original paper [66].
Figure 2 shows some non-cherry-picked generated samples from a single-step generator trained with Diff-Instruct on the FFHQ [28], ImageNet [11], and the CIFAR10 [36] datasets of the resolution of  $64 \times 64$ . In conclusion, our Diff-Instruct can achieve competitive distillation performance under the most challenging conditions with no synthetic or real datasets. We put more discussions and analyses in the Appendix B.1.
Remark 4.1. In Table 1 and Table 2, PD, CD, and Diff-Instruct all use the EDM teacher, and the same UNet student with the same architecture as the teacher model (The author of consistency distillation re-implemented the PD for EDM, so the reported FID for PD is lower than that in PD's original paper). The Diff-Instruct generator uses the same UNet architecture as the teacher diffusion model, so the number of sampling steps (NFE) represents its inference time costs. As we show in the upper part of Table 2, sampling from our learned one-step generator with 1 NFE results in an FID of 4.19, which is significantly better than its teachers with 10 NFEs with an FID of 15.56. So we conclude that the one-step generator trained with Diff-Instruct achieves at least  $10+$  times acceleration (more efficient) than its teacher diffusion model.
Fast convergence speed. Another advantage of applying Diff-Instruct for diffusion distillation is the fast convergence speed. We empirically find that Diff-Instruct has a much faster convergence speed than other distillation methods and has a tolerance for a large learning rate for optimization. In Figure 3, we show the convergence of FID with respect to the optimization iterations of the generator trained on an unconditional DM on the CIFAR10 dataset. We set the optimization step size of Diff-Instruct to be  $1e - 4$  and the FID of the generator trained with Diff-Instruct converges fast within 7k iterations. However, the FID of the distilled diffusion model with consistency distillation algorithms does not converge with less than 7k iterations. One possible reason for the fast convergence speed is that Diff-Instruct's student model is a one-step generator that does not need to take multiple-time indexes in contrast to student models of other distillation methods such as CD and PD. This makes Diff-Instruct efficient when distillation without the need for learning at multiple time levels.
4.2 Improving generative adversarial networks
Another application of Diff-Instruct is to improve the generator of GAN models that are pre-trained with adversarial training 3.3.2.
Experiment settings. We take the pre-trained EDM model [32] on the CIFAR10 datasets as the instructor and the pre-trained StyleGAN-2 [29] models that are assumed to converge with adversarial training under different settings (conditional or unconditional with different data augmentation strategies). Our goal is to use pre-trained DMs to further improve pre-trained generators. We initialize the generator in the Diff-Instruct algorithm with the pre-trained StyleGAN-2 generator and initialize DMs for implicit distributions with the pre-trained EDM models. We then use Diff-Instruct to update the generator. We put more details in Appendix B.2.
Performance. As shown in Table 3 and 4, our Diff-Instruct can consistently improve the pre-trained generator's performance in terms of FID. More precisely, the FID of the pre-trained StyleGAN-2 with Adaptive Data Augmentation (ADA) is improved from 2.42 to 2.27 for the conditional setting and
Table 3: Class-conditional sample quality on CIFAR10 through GAN models. † Models that we implemented.
METHOD	FID (↓)	IS (↑)
BigGAN [5]	14.73	9.22
BigGAN+Tune [5]	8.47	9.07 ± 0.13
MultiHinge [33]	6.40	9.58 ± 0.09
FQ-GAN [81]	5.59	8.48 ± 0.09
Stylegan2 [30]	6.96	9.53 ± 0.06
Stylegan2†	7.01	9.23 ± 0.07
Stylegan2† + DI	6.62	9.40 ± 0.06
Stylegan2+ADA [29]	3.49	10.24 ± 0.07
Stylegan2+ADA+Tune [29]	2.42	10.14 ± 0.09
Stylegan2+ADA+Tune + DI	2.27	10.11 ± 0.10
Table 4: Unconditional sample quality on CIFAR10 through GAN models. † Models that we implemented.
METHOD	FID (↓)	IS (↑)
SNGAN [49]	21.70	8.22
ProGAN [17]	15.52	8.56 ± 0.06
AutoGAN [18]	12.42	8.55 ± 0.10
SNGAN+DGflow [1]	9.35	9.62
TransGAN [27]	9.02	9.26
StyleGAN2 [30]	8.32	9.21 ± 0.09
StyleGAN2†	8.21	9.09 ± 0.09
StyleGAN2† + DI	7.56	9.16 ± 0.09
StyleGAN2+ADA [29]	5.33	10.02 ± 0.07
StyleGAN2+ADA+Tune [29]	2.92	9.83 ± 0.04
StyleGAN2+ADA+Tune + DI	2.71	9.86 ± 0.04
from 2.92 to 2.71 for the unconditional setting. The experiment shows that Diff-Instruct is able to inject the knowledge of pre-trained diffusion models to enhance the generators further.
The results demonstrate that Diff-Instruct is a powerful method capable of improving existing GAN models that are supposed to converge with adversarial training. There are two possible reasons for such improvements. First, our Diff-Instruct utilizes well-trained diffusion models to supervise the generator. For instance, on the CIFAR10 dataset with conditional labels, the teacher EDM model can achieve the FID of 1.79, which is significantly better than StyleGAN2 with an FID of 2.42. Second, Diff-Instruct takes diffused data into account when minimizing the IKL, overcoming the potential degeneration issues of the divergences that adversarial training intends to minimize.
This work presents a novel learning paradigm, Diff-Instruct, which is to our best knowledge, the first method that enables knowledge transfer from pre-trained diffusion models into generic generators in a data-free manner. The theoretical foundations and practical methods introduced in this work hold promise for advancing the utilization of diffusion generative models and implicit models across various domains and applications.
Nonetheless, Diff-Instruct has its limitations that call for further research along this line. First, with the abundance of powerful pre-trained DMs with diverse expertise, leveraging multiple models as instructors is another promising direction that is not investigated in this work. Second, even though data-free is a feature of our method, utilizing real data can potentially boost the learning process. The potential benefits of incorporating both Diff-Instruct and training data have not been explored yet. Lastly, in the extreme case where we have only data and no pre-trained DMs, our DI framework can still be adapted. Potentially we can train a teacher diffusion model with data, and concurrently, use it to instruct the student model. This indirect way of training may enable other generative models such as GANs, to enjoy the benefits of diffusion models, e.g., ease of training, ability to scale, and high sample quality, etc.
W. Luo and Z. Zhang have been supported by the Beijing Natural Science Foundation (Z190001) and the National Natural Science Foundation of China (No. 12271011).
[70] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi. Learning fast samplers for diffusion models by differentiating through sample quality. In International Conference on Learning Representations, 2022.
[71] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvjiotham, and Ali Taylan Cemgil. A fine-grained analysis on distribution shift. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Dl4LetuLdyK.
[72] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pages 23965-23998. PMLR, 2022.
[73] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In International Conference on Learning Representations, 2021.
[74] Yilun Xu, Ziming Liu, Max Tegmark, and Tommi S. Jaakkola. Poisson flow generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=voV_TRqcWh.
[75] Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6199-6203. IEEE, 2020.
[76] Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang. Midinet: A convolutional generative adversarial network for symbolic-domain music generation. arXiv preprint arXiv:1703.10847, 2017.
[77] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In AAAI Conference on Artificial Intelligence, 2017.
[78] Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204.13902, 2022.
[79] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[80] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. arXiv preprint arXiv:2302.04867, 2023.
[81] Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization improves gan training. arXiv preprint arXiv:2004.02088, 2020.
[82] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadeneshli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.13449, 2022.
[83] Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion-based adversarial auto-encoders. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=HDxgaKk9561.
A.1 Robustness of Integral KL divergence
One of the benefits of using the Integral KL divergence over the traditional KL divergence is its robustness to misaligned density support. To illustrate this advantage, we consider a well-known example in [2]. Let  $z$  be a random variable following the uniform distribution on the unit interval [0,1]. Consider  $\mathbb{P}_0$  to be the distribution of  $(0,z) \in \mathbb{R}^2$ . Now let  $g_{\theta}(z) = (\theta ,z)$ , where  $\theta$  is a single real parameter. The density of  $\mathbb{P}_0$  and  $\mathbb{P}_{\theta}$  are  $p_0(\pmb {x},z) = \mathbb{I}_{\pmb{x} = 0}(\pmb {x})\mathbb{I}_{[0,1]}(\pmb {z})$  and  $p_{\theta}(\pmb {x},z) = \mathbb{I}_{\theta}(\pmb {x})\mathbb{I}_{[0,1]}(\pmb {z})$ . Since for each  $\theta \neq 0$ , the support of  $p_{\theta}$  and  $p_0$  does not intersect, the KL divergence between  $\mathbb{P}_{\theta}$  and  $\mathbb{P}_0$  is ill-defined with
$$
\mathcal {D} _ {K L} \left(\mathbb {P} _ {\theta}, \mathbb {P} _ {0}\right) = \left\{ \begin{array}{l l} + \infty & \theta \neq 0; \\ 0 & \theta = 0. \end{array} \right. \tag {A.1}
$$
The same is also true for the Jensen-Shannon divergence where
$$
\mathcal {D} _ {J S} \left(\mathbb {P} _ {\theta}, \mathbb {P} _ {0}\right) = \left\{ \begin{array}{l l} \log 2 & \theta \neq 0; \\ 0 & \theta = 0. \end{array} \right. \tag {A.2}
$$
So minimizing the KL divergence with a gradient-based algorithm does not lead the generator to converge to the correct parameter  $\theta = 0$ . However, IKL provides a finite and reliable objective for training the generator. More precisely, considering a simple diffusion
$$
\mathrm {d} \boldsymbol {x} _ {t} = \mathrm {d} \boldsymbol {w} _ {t}. \tag {A.3}
$$
The marginal distribution of  $(x,z)$  under diffusion (A.3) initialized with  $\mathbb{P}_0$  and  $\mathbb{P}_{\theta}$  writes
$$
p _ {0} ^ {(t)} (x, z) = \mathcal {N} (x; 0, t) \int_ {0} ^ {1} \mathcal {N} (z; s, t) \mathrm {d} s,
$$
$$
p _ {\theta} ^ {(t)} (x, z) = \mathcal {N} (x; \theta , t) \int_ {0} ^ {1} \mathcal {N} (z; s, t) \mathrm {d} s,
$$
which are defined on  $(x,z)\in \mathbb{R}^2$ . The notation  $\mathcal{N}(x;\mu ,\sigma^2)$  represents the density function of Gaussian distribution with mean  $\mu$  and variance  $\sigma^2$ . The IKL divergence with weight function  $w(t)$  thus has the expression