SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathcal {D} _ {I K L} (\mathbb {P} _ {\theta}, \mathbb {P} _ {0}) = \int_ {t = 0} ^ {\infty} w (t) \left[ \mathbb {E} _ {(x, z) \sim p _ {\theta} ^ {(t)} (x, z)} \log \frac {p _ {\theta} ^ {(t)} (x , z)}{p _ {\theta} ^ {(0)} (x , z)} \right] d t \\ = \int_ {t = 0} ^ {\infty} w (t) \left[ \mathbb {E} _ {(x, z) \sim p _ {\theta} ^ {(t)} (x, z)} \log \frac {\mathcal {N} (x ; \theta , t)}{\mathcal {N} (x ; 0 , t)} \right] d t \\ = \int_ {t = 0} ^ {\infty} w (t) \left[ \mathbb {E} _ {x \sim \mathcal {N} (x; \theta , t)} \log \frac {\mathcal {N} (x ; \theta , t)}{\mathcal {N} (x ; 0 , t)} \right] d t \tag {A.4} \\ = \int_ {t = 0} ^ {\infty} w (t) \left[ \mathbb {E} _ {x \sim \mathcal {N} (x; \theta , t)} \frac {1}{2 t} \left[ x ^ {2} - (x - \theta) ^ {2} \right] \right] d t \\ = \int_ {t = 0} ^ {\infty} w (t) \left[ \mathbb {E} _ {x \sim \mathcal {N} (x; \theta , t)} \frac {1}{2 t} [ 2 \theta x - \theta^ {2} ] \right] d t \\ = \theta^ {2} \left[ \int_ {0} ^ {\infty} \frac {w (t)}{2 t} \mathrm {d} t \right], \theta \in \mathbb {R}. \\ \end{array}
$$
By properly choosing the weighting function  $w(t)$ ,  $\mathcal{D}_{IKL}$  is finite as long as  $\int_0^T\frac{w(t)}{2t}\mathrm{d}t$  is finite. For instance,  $w(t) = 1 / t$  if  $t\geq 1$  and  $w(t) = t$  if  $t\leq 1$  as a simple choice<sup>4</sup>. The IKL divergence in (A.4) is a differentiable quadratic function of parameter  $\theta$  with a single minima  $\theta = 0$  which lead to the  $\mathcal{D}_{IKL}(\mathbb{P}_\theta ,\mathbb{P}_0) = 0$ .
Table 5 shows a summary of the comparison among IKL divergence, KL divergence and the Wasserstein distance between  $\mathbb{P}_{\theta}$  and  $\mathbb{P}_0$ . Our IKL is more suitable for learning  $\theta$  with gradient-based optimization algorithms.
In practice when distilling from pre-trained diffusion models, we use the same weighting function for training pre-trained diffusion models for Diff-Instruct, which is also inverted U-shaped.
Table 5: Comparison of divergence property against IKL.
Divergence	distance	smooth
IKL (ours)	\( \propto \theta^2 \)	✓
KL	+\(\infty\)(A.1)	✘
Wasserstein	\( \propto |\theta| [2] \)	✘
Jensen-Shannon	log2 (A.2)	✘
A.2 Proof of Theorem 3.3
Proof. Recall the definition of  $q^{(t)}$ , the sample is obtained by  $\pmb{x}_0 = g_\theta(\pmb{z}), \pmb{z} \sim p_z$ , and  $\pmb{x}_t | \pmb{x}_0 \sim p_t(\pmb{x}_t | \pmb{x}_0)$  according to forward SDE (2.1). Since the solution of forward, SDE is uniquely determined by the initial point  $\pmb{x}_0$  and a trajectory of Wiener process  $\pmb{w}_{t \in [0,T]}$ , we slightly abuse the notation and let  $\pmb{x}_t = \mathcal{F}(g_\theta(\pmb{z}), \pmb{w})$  to represent the solution of  $\pmb{x}_t$  generated by  $\pmb{x}_0$  and  $\pmb{w}$ . We let  $\pmb{w}_{[0,1]} \sim \mathbb{P}_{\pmb{w}}$  to demonstrate a trajectory from the Wiener process where  $\mathbb{P}_{\pmb{w}}$  represents the path measure of Wiener process on  $t \in [0,T]$ . There are two terms that contain the generator's parameter  $\theta$ . The term  $\pmb{x}_t$  contains parameter through  $\pmb{x}_0 = g_\theta(\pmb{z}), \pmb{z} \sim p_z$ . The marginal density  $q^{(t)}$  also contains parameter  $\theta$  implicitly since  $q^{(t)}$  is initialized with  $q^{(0)}$  which is generated by the generator. To demonstrate the parameter dependence, we may use  $q_{\theta}^{(t)}$  to represent  $q^{(t)}$ .
The  $p^{(t)}$  is defined through the pre-trained diffusion models with score functions  $s_{p^{(t)}}$ . The IKL divergence between  $q^{(t)}$  and  $p^{(t)}$  is defined with,
$$
\begin{array}{l} \mathcal {D} _ {I K L} ^ {[ 0, T ]} \left(q _ {\theta} ^ {(0)}, p ^ {(0)}\right) := \int_ {t = 0} ^ {T} w (t) \mathcal {D} _ {K L} \left(q _ {\theta} ^ {(t)}, p ^ {(t)}\right) \mathrm {d} t \\ = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {x} _ {t} \sim q _ {\theta} ^ {(t)}} \left[ \log \frac {q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t})}{p ^ {(t)} (\boldsymbol {x} _ {t})} \right] \mathrm {d} t \\ = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\substack {\boldsymbol {z} \sim p _ {z}, \\ \boldsymbol {w} \sim \mathbb {P} _ {\boldsymbol {w}}}} \left[ \log \frac {q _ {\theta} ^ {(t)} \left(\mathcal {F} \left(g _ {\theta} (\boldsymbol {z}) , \boldsymbol {w}\right)\right)}{p ^ {(t)} \left(\mathcal {F} \left(g _ {\theta} (\boldsymbol {z}), \boldsymbol {w}\right)\right)} \right] d t \tag{A.5} \\ \end{array}
$$
Taking the  $\theta$  gradient of IKL (A.5), we have
$$
\begin{array}{l} \frac {\partial}{\partial \theta} \mathcal {D} _ {I K L} ^ {[ 0, T ]} (q _ {\theta} ^ {(t)}, p ^ {(t)}) \\ = \frac {\partial}{\partial \theta} \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {w} \sim \mathbb {P} _ {\boldsymbol {w}}} ^ {\boldsymbol {z} \sim p _ {z},} \left[ \log \frac {q _ {\theta} ^ {(t)} (\mathcal {F} (g _ {\theta} (\boldsymbol {z}) , \boldsymbol {w}))}{p ^ {(t)} (\mathcal {F} (g _ {\theta} (\boldsymbol {z}), \boldsymbol {w})))} \right] \\ = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {w} \sim \mathbb {P} _ {\boldsymbol {w}}} ^ {\boldsymbol {z} \sim_ {p _ {z}},} \frac {\partial}{\partial \theta} \left[ \log \frac {q _ {\theta} ^ {(t)} (\mathcal {F} (g _ {\theta} (\boldsymbol {z}) , \boldsymbol {w}))}{p ^ {(t)} (\mathcal {F} (g _ {\theta} (\boldsymbol {z}), \boldsymbol {w})))} \right] \\ = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\substack {\boldsymbol {z} \sim p _ {z}, \\ \boldsymbol {w} \sim \mathbb {P} _ {\boldsymbol{w}}}} \nabla_ {\boldsymbol {x} _ {t}} \left[ \log \frac {q _ {\theta} ^ {(t)} (\mathcal {F} (g _ {\theta} (\boldsymbol {z}) , \boldsymbol {w}))}{p ^ {(t)} (\mathcal {F} (g _ {\theta} (\boldsymbol {z}), \boldsymbol {w})))} \right] \frac {\partial \mathcal {F} (g _ {\theta} (\boldsymbol {z}) , \boldsymbol {w})}{\partial \theta} \\ + \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {w} \sim \mathbb {P} _ {\boldsymbol {w}}} \frac {\partial}{\partial \theta} \log q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) | _ {\boldsymbol {x} _ {t} = \mathcal {F} (g _ {\theta} (\boldsymbol {z}), \boldsymbol {w})} \\ = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\substack {\boldsymbol {z} \sim p _ {z}, \boldsymbol {w} \sim \mathbb {P} _ {\boldsymbol {w}}, \boldsymbol {x} _ {0} = g _ {\theta} (\boldsymbol {z}) \\ \boldsymbol {x} _ {t} = \mathcal {F} (\boldsymbol {x} _ {0}, \boldsymbol {w})}} \nabla_ {\boldsymbol {x} _ {t}} \left[ \log \frac {q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t})}{p ^ {(t)} (\boldsymbol {x} _ {t}))} \right] \frac {\partial \boldsymbol {x} _ {t}}{\partial \theta} + \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {x} _ {t} \sim p _ {\theta} ^ {(t)}} \frac {\partial}{\partial \theta} \log q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) \\ = A + B. \tag {A.6} \\ \end{array}
$$
The term  $A$  in equation (A.6) writes
$$
A = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {x} _ {t} \sim q ^ {(t)}} \left[ \boldsymbol {s} _ {q ^ {(t)}} (\boldsymbol {x} _ {t}) - \boldsymbol {s} _ {p ^ {(t)}} (\boldsymbol {x} _ {t}) \right] \frac {\partial \boldsymbol {x} _ {t}}{\partial \theta} \tag {A.7}
$$
We show that the term  $B$  in equation (A.6) vanishes.
$$
\begin{array}{l} B = \int_ {t = 0} ^ {T} w (t) \mathbb {E} _ {\boldsymbol {x} _ {t} \sim q _ {\theta} ^ {(t)}} \frac {\partial}{\partial \theta} \log q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) \\ = \int_ {t = 0} ^ {T} w (t) \int \frac {1}{p _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t})} \frac {\partial}{\partial \theta} q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) d \boldsymbol {x} _ {t} \\ = \int_ {t = 0} ^ {T} w (t) \int \frac {\partial}{\partial \theta} q _ {\theta} ^ {(t)} (\boldsymbol {x} _ {t}) \mathrm {d} \boldsymbol {x} _ {t} \\ = \int_ {t = 0} ^ {T} w (t) \frac {\partial}{\partial \theta} \int q _ {\theta} ^ {(t)} \left(\boldsymbol {x} _ {t}\right) \mathrm {d} \boldsymbol {x} _ {t} (A.8) \\ = \int_ {t = 0} ^ {T} w (t) \frac {\partial}{\partial \theta} \mathbf {1} d \mathbf {x} _ {t} \\ = 0 (A.9) \\ \end{array}
$$
The equality (A.8) holds if function  $q_{\theta}^{(t)}(\pmb{x})$  satisfies the conditions (1).  $q_{\theta}^{(t)}(\pmb{x})$  is Lebesgue integrable for  $\pmb{x}$  with each  $\theta$ ; (2). For almost all  $\pmb{x} \in \mathbb{R}^D$ , the partial derivative  $\partial q_{\theta}^{(t)}(\pmb{x}) / \partial \theta$  exists for all  $\theta \in \Theta$ . (3) there exists an integrable function  $h(.): \mathbb{R}^D \to \mathbb{R}$ , such that  $q_{\theta}^{(t)}(\pmb{x}) \leq h(\pmb{x})$  for all  $\pmb{x}$  in its domain. Then the derivative w.r.t  $\theta$  can be exchanged with the integral over  $\pmb{x}$ , i.e.
$$
\int \frac {\partial}{\partial \theta} q _ {\theta} ^ {(t)} (\boldsymbol {x}) d \boldsymbol {x} = \frac {\partial}{\partial \theta} \int q _ {\theta} ^ {(t)} (\boldsymbol {x}) d \boldsymbol {x}.
$$
Remark A.1. In practice, most commonly used forward diffusion processes can be expressed as a form of scale and noise addition:
$$
\boldsymbol {x} _ {t} = \alpha (t) \boldsymbol {x} _ {0} + \beta (t) \epsilon , \quad \epsilon \sim \mathcal {N} (\epsilon ; 0, \mathbf {I}). \tag {A.10}
$$
So the term  $\mathbf{z} \sim p_z$ ,  $\mathbf{w} \sim \mathbb{P}_{\mathbf{w}}$ ,  $\mathbf{x}_t = \mathcal{F}(\mathbf{x}_0, \mathbf{w})$  in equation (A.5) can be instantiated as  $\mathbf{z} \sim p_z$ ,  $\epsilon \sim \mathcal{N}(\epsilon; 0, I)$ ,  $\mathbf{x}_t = \alpha(t)\mathbf{x}_0 + \beta(t)\epsilon$ .
A.3 Proof of Corollary 3.4
Proof. Since the  $q(\pmb{x}_0) = \delta_{g(\theta)}(\pmb{x}_0)$ , thus the conditional distribution and marginal distribution coincide, i.e.
$$
q ^ {(t)} (\boldsymbol {x} _ {t}) = \int p _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0}) q ^ {(0)} (\boldsymbol {x} _ {0}) \mathrm {d} \boldsymbol {x} _ {0} = p _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0}) \mathbb {I} _ {g (\theta)} (\boldsymbol {x} _ {0}) = p _ {t} (\boldsymbol {x} _ {t} | g (\theta))
$$
So the marginal score function writes
$$
\boldsymbol {s} _ {q ^ {(t)}} (\boldsymbol {x} _ {t}) := \nabla_ {\boldsymbol {x} _ {t}} \log q ^ {(t)} (\boldsymbol {x} _ {t}) = \nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} (\boldsymbol {x} _ {t} | g (\theta))
$$
So the gradient formula (3.2) turns to
$$
\operatorname {Grad}(\theta) = \int_{t = 0}^{T}w(t)\mathbb{E}_{\substack{\boldsymbol{x}_{0} = g(\theta),\\ \boldsymbol{x}_{t}|\boldsymbol{x}_{0}\sim p_{t}(\boldsymbol{x}_{t}|\boldsymbol{x}_{0})}}\left[\nabla_{\boldsymbol{x}_{t}}\log p_{t}(\boldsymbol{x}_{t}|\boldsymbol{x}_{0}) - \boldsymbol{s}_{p^{(t)}}(\boldsymbol{x}_{t})\right]\frac{\partial\boldsymbol{x}_{t}}{\partial\theta}\mathrm{d}t.
$$
If we define a  $\epsilon$ -network  $\epsilon_p(x, t) := -s_{p^{(t)}}(\boldsymbol{x}_t) / \sigma(t)$  and consider the forward diffusion  $p_t(\boldsymbol{x}_t | \boldsymbol{x}_0) = \mathcal{N}(\boldsymbol{x}_t; \alpha(t) \boldsymbol{x}_0, \sigma(t) \boldsymbol{I})$ , the forward diffusion can be implemented with  $\boldsymbol{x}_t = \alpha(t) \boldsymbol{x}_0 + \sigma(t) \epsilon, \epsilon \sim \mathcal{N}(\epsilon; \mathbf{0}, \boldsymbol{I})$ , then the objective turns to
$$
\operatorname {G r a d} ^ {(S D S)} (\theta) = \int_ {0} ^ {T} \frac {w (t)}{\sigma (t)} \mathbb {E} _ {\substack {\boldsymbol {x} _ {0} = g _ {\theta} (\boldsymbol {z} _ {0}), \epsilon \sim \mathcal {N} (\boldsymbol {0}, \mathbf {I}), \\ \boldsymbol {x} _ {t} = \alpha (t) \boldsymbol {x} _ {0} + \sigma (t) \epsilon}} \left[ \epsilon_ {p} (\boldsymbol {x} _ {t}, t) - \epsilon \right] \frac {\partial \boldsymbol {x} _ {t}}{\partial \theta} \mathrm {d} t, \tag{A.11}
$$
which recovers the SDS gradient estimation proposed in DreamFusion [54]. In summary, we have shown that our Diff-Instruct's gradient is equivalent to the gradient formula of score distillation sampling (SDS) when the generator outputs a single data that is differentiable to the generator's parameters. This is not a coincidence, as a NeRF model can be viewed as a generator that outputs a Dirac's Delta distribution when the view direction is fixed. Therefore, using SDS to learn a text-conditioned NeRF model is essentially an application of using an approximated version of Diff-Instruct to distill a pre-trained text-to-2D diffusion model in order to obtain a NeRF.
A.4 Proof of Corollary 3.5
Proof. Following the same notation as in section 2, consider the adversarial training that minimizes the KL divergence as we introduced in Section 2. The learning objective is
$$
\mathcal {L} ^ {(K L)} (\theta) = \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \left[ \log \frac {1 - h \left(g _ {\theta} (\boldsymbol {z})\right)}{h \left(g _ {\theta} (\boldsymbol {z})\right)} \right] \tag {A.12}
$$
Assume the discriminator  $h(.)$  is optimal, then
$$
h (\boldsymbol {x}) = \frac {p _ {d} (\boldsymbol {x})}{p _ {d} (\boldsymbol {x}) + p _ {g} (\boldsymbol {x})}, a n d \log \frac {1 - h (\boldsymbol {x})}{h (\boldsymbol {x})} = \log \frac {p _ {d} (\boldsymbol {x})}{p _ {g} (\boldsymbol {x})}
$$
Then gradient for the generator parameter  $\theta$  turn to
$$
\begin{array}{l} \frac {\partial}{\partial \theta} \mathcal {L} ^ {(K L)} (\theta) = \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \nabla_ {\boldsymbol {x}} \left(\log \frac {1 - h (\boldsymbol {x})}{h (\boldsymbol {x})}\right) | _ {\boldsymbol {x} = g _ {\theta} (\boldsymbol {z})} \frac {\partial g _ {\theta} (\boldsymbol {z})}{\partial \theta} \\ = \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \nabla_ {\boldsymbol {x}} \left(\log \frac {p _ {d} (\boldsymbol {x})}{p _ {g} (\boldsymbol {x})}\right) | _ {\boldsymbol {x} = g _ {\theta} (\boldsymbol {z})} \frac {\partial g _ {\theta} (\boldsymbol {z})}{\partial \theta} \\ = \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \left[ \nabla_ {\boldsymbol {x}} \log p _ {d} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log p _ {g} (\boldsymbol {x}) \right] | _ {\boldsymbol {x} = g _ {\theta} (\boldsymbol {z})} \frac {\partial g _ {\theta} (\boldsymbol {z})}{\partial \theta} \\ = \underset {\boldsymbol {x} = g _ {\theta} (\boldsymbol {z})} {\mathbb {E}} \left[ \nabla_ {\boldsymbol {x}} \mathrm {l o g} p _ {d} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \mathrm {l o g} p _ {g} (\boldsymbol {x}) \right] \frac {\partial \boldsymbol {x}}{\partial \theta} \\ = \mathbb {E} _ {\substack {\boldsymbol {z} \sim p _ {z}, \\ \boldsymbol {x} = g _ {\theta} (\boldsymbol {z})}} \left[ \boldsymbol {s} _ {d} (\boldsymbol {x}) - \boldsymbol {s} _ {g} (\boldsymbol {x}) \right] \frac {\partial \boldsymbol {x}}{\partial \theta} \tag{A.13} \\ \end{array}
$$
where  $s_g(\boldsymbol{x}) = \nabla_{\boldsymbol{x}} \log p_g(\boldsymbol{x})$  and  $s_d(\boldsymbol{x}) = \nabla_{\boldsymbol{x}} \log p_d(\boldsymbol{x})$  denote the score function of the generator and the data distribution. The equation (A.13) shows that the adversarial training that minimizes the KL divergence is equivalent to the gradient formula (3.2) with a special weight function  $w(0) = 1$  and  $w(t) > 0$  for all  $t > 0$ , if the discriminator can be trained to be optimal.
A.4.1 Comparison of Distillation Methods
Table 6: Comparison of Diffusion Distillation Methods. The term efficiency represents the training efficiency of DMs. flexibility means whether the student model needs to have the same in-out dimensions.
Method	data-free	flexibility	efficiency
ReFlow[40]	✓	✗	low
DFNO[82]	✓	✗	low
KD[42]	✓	✗	low
CD[66]	✗	✗	medium
PD[61]	✓	✗	low
DI (ours)	✓	✓	high
B More on experiments
To demonstrate the efficacy of our proposed Diff-Instruct, we choose the state-of-the-art EDMs [32] as instructors, which have achieved state-of-the-art generative performance on several benchmarks
Table 7: Hyperparameters used for Diff-Instruct for Diffusion Distillation
Hyperparameter	CIFAR-10 (Uncond)	ImageNet 64×64	CIFAR-10 (Cond)
DM sφ	Generator gθ	DM sφ	Generator gθ	DM sφ	Generator gθ
Learning rate	1e-5	1e-5	1e-5	1e-5	1e-5	1e-5
Batch size	64	64	96	96	64	64
σ(t*)	2.5	2.5	5.0	5.0	2.5	2.5
Adam β0	0.0	0.0	0.0	0.0	0.0	0.0
Adam β1	0.99	0.99	0.99	0.99	0.99	0.99
Training iterations	100k	100k	50k	50k	100k	100k
Number of GPUs	4	4	8	8	4	4
Figure 4: Left: Demonstration of the training process for Diff-Instruct for diffusion distillation. Generated samples with the same latent vectors and different generator weights during training are put from the top to the bottom. Right: generated samples from a one-step generative model distilled from pre-trained class-conditional EMD model on the CIFAR10 dataset.
such as CIFAR10 and ImageNet  $64 \times 64$ . The EDM model depends on the diffusion process
$$
d \boldsymbol {x} _ {t} = G (t) d \boldsymbol {w} _ {t}, t \in [ 0, T ]. \tag {B.1}
$$
Samples from the forward process (B.1) can be generated by adding random noise to the output of the generator function, i.e.,  $\boldsymbol{x}_t = \boldsymbol{x}_0 + \sigma(t)\epsilon$  where  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  is a Gaussian vector and  $\sigma(t) \coloneqq \sqrt{\int_0^t G^2(s)ds}$  is a function with explicit expressions. We download the pre-trained model checkpoints from the official website<sup>5</sup> and consider transferring their knowledge to implicit generative models, specifically UNet and StyleGAN as generators.
We calculate the FID and IS in the same way as the StyleGAN2-ADA codebase. For the ImageNet  $64 \times 64$  dataset, we use the same pre-processing scripts as the EDM model on ImageNet  $64 \times 64$  dataset.
B.1 Detailed experimental settings of diffusion distillation
When the downstream generative model is a UNet generator, our DI provides another way for diffusion distillation, directly competing with progressive distillation [61] or consistency distillation [66].
We initialize the diffusion model for the implicit distribution  $s_{\phi}$  with the weight parameters of pre-trained DM. The initialization of the generator is put in the following paragraph. With the pre-trained DM, initialized generator, and the initialized DM for implicit distribution, we use our Diff-Instruct algorithm 1 to update the DM for implicit distribution and the generator's parameters. We use the Adam optimizer for both the DM and the generator. For the generator, we use the same exponential moving average (EMA) technique as the EDM model's training scripts.
To make a fair comparison, we do not compare approaches that involve other components such as classifier guidance or additional architectures for the generation as [82] because these models have significantly larger model sizes for inference.
$^{5}$ https://github.com/NVlabs/edm
<sup>6</sup>https://github.com/NVlabs/stylegan2-ada-pytorch
Table 8: Comparison of Memory Costs and Wall-Clock Timing of CD and Diff-Instruct. The Peak GPU-Memo shows the Maximum observed GPU memory caused by the Tensor and Computational Graph of the host GPU. Sec-per-K Iterations report the wall-clock time for each training iteration. For Diff-Instruct, each iteration consists of two stages as in Algorithm 1 in the main text. Test environment: PyTorch 1.12.1 and Torchvision 0.13.1, and Torch.distributed.parallel on 2 V100 GPUs.
Method	Peak GPU-Memo(GB)	Peak CPU-Memo(GB)	Sec-per-K Iterations
CD	9.55	2.75	0.0489
Diff-Instruct	10.40	2.78	0.0728
Initialization of the generator. When the generator is chosen to have the same architecture as the pre-trained diffusion models (i.e. UNet in most cases), we can initialize the generator with a special method with the score network of the pre-trained DM. Taking the forward diffusion process (B.1) as an instance. The time-indexed score functions explicitly define a data-prediction transform through Tweedie's formula
$$
\widehat {\boldsymbol {x}} _ {0} := \boldsymbol {x} _ {t} + \sigma (t) ^ {2} \nabla_ {\boldsymbol {x} _ {t}} \log q ^ {(t)} (\boldsymbol {x} _ {t}) \tag {B.2}
$$
This formula tells that the marginal score functions can be converted to a data-prediction transform which can transform noisy data to clean data. Motivated by this property, we initialize the implicit generator  $g_{\theta}$  for Diff-Instruct in Algorithm 1 via a score network-induced data-prediction transform of the teacher diffusion models at some fixed time  $t^{*}$ :
$$
\boldsymbol {x} = \boldsymbol {z} + \sigma^ {2} \left(t ^ {*}\right) \boldsymbol {s} _ {p \left(t ^ {*}\right)} (\boldsymbol {z}). \tag {B.3}
$$
where  $z \sim \mathcal{N}(z; 0, \sigma^2(t^*) I)$  is the latent vector. This generator takes a Gaussian noise with the variance  $\sigma(t^*)$  and zero mean as an input latent vector.
We find that each noise level  $\sigma(t^{*})$  can give a comparable initialization, so we roughly selected an  $\sigma(t^{*})$  for different datasets. We put detailed hyper-parameters for distilling in Table 7. The left hand of Figure 4 gives a demonstration of the samples with the same latent vectors from different generators during training. The top line is the initialized generator (i.e. modified from pre-trained diffusion models). During the training, the initialized generator is trained to generate high-quality samples. The right hand of Figure 4 shows class-conditioned samples from a one-step model (generator) distilled from a class-conditional EDM model on the CIFAR10 dataset.
Comparison of Computational Costs The Diff-Instruct algorithm involves training an additional auxiliary diffusion model during the training phases. It is worth noting that although this auxiliary diffusion model brings additional memory cost, this additional memory cost is very limited because the memory bottleneck of training lies in the computational graph of the backpropagation, instead of only saving one more model.
In the Diff-Instruct algorithm, the model and the generator are updated alternatively, which means that the other model's parameters are fixed and do not participate in back-propagation when one model is being updated. So the memory cost for back-propagating through the computational graph is almost the same as one model.
To quantitatively measure how much additional computational costs are brought in, we run an experiment to compare the computational and memory costs of Diff-Instruct with our baseline method, the Consistency Distillation, in 8. The test was run on 2 Nvidia V100 GPUs with 128 batch size and PyTorch distributed data-parallel mechanism.
The result shows that Diff-Instruct brings in minor additional memory costs than CD (10.40 over 9.55). This is because the Diff-Instruct only needs additional GPU memory to save the auxiliary model  $s_{\phi}$ . But the  $s_{\phi}$  and generator  $g_{\theta}$  are updated alternatively, so their computational graph does not interact. As a result, the memory bottleneck caused by computational graph and back-propagation does not bring more costs to Diff-Instruct.
As for the wall-clock time for 1K iterations, we see that Diff-Instruct costs 0.0728 seconds, while the CD costs 0.0489 seconds. This is because each iteration of Diff-Instruct consists of two alternate steps as we show in Algorithm 1. Overall, the Diff-Instruct costs almost the same GPU and CPU memory as the baseline CD, but about 1.5 times wall-clock time than the CD for each iteration.
Table 9: Hyperparameters used for Diff-Instruct for GAN Improvement on CIFAR10 dataset under different settings.
Hyperparameter	StyleGAN-2 + Cond	StyleGAN-2 + Uncond	StyleGAN-2-ADA + Cond	StyleGAN-2-ADA + Uncond
DM sφ	Generator gθ	DM sφ	Generator gθ	DM sφ	Generator gθ	DM sφ	Generator gθ
Learning rate	2e-7	2e-7	1.5e-6	1.5e-6	1.5e-6	1.5e-6	1.5e-6	1.5e-6
Batch size	512	512	512	512	512	512	512	512
Adam β0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Adam β1	0.99	0.99	0.99	0.99	0.99	0.99	0.99	0.99
Training iterations	16k	16k	6k	6k	9k	9k	10k	10k
Number of GPUs	4	4	4	4	4	4	4	4
Figure 5: Generated samples from improved generators with Diff-Instruct in Table 3 and 1. The FID for generated samples from left to right is 2.27, 2.71, 6.62, and 7.56.
B.2 Detailed experimental settings of GAN improvement
This experiment aims to show the power of Diff-Instruct to improve GAN's generator under different settings by transferring knowledge from pre-trained DMs.
Experiment settings. The StyleGAN-2 model is a competitive GAN model on benchmarking datasets such as the CIFAR10 dataset, so we take StyleGAN-2 as the representative GAN model. We take the pre-trained EDM models with VP architecture  $^{7}$  on the CIFAR10 datasets as pre-trained diffusion models.