SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 4.2. The ground truth reward  $f^{*}(x) = g^{*}(x_{\parallel}) - h^{*}(x_{\perp})$ , where  $g^{*}(x_{\parallel}) = (\theta^{*})^{\top}x_{\parallel}$  where  $\theta^{*} = A\beta^{*}$  for some  $\beta^{*}\in \mathbb{R}^{d}$  and  $\| \theta^{*}\|_{2} = \| \beta^{*}\|_{2} = 1$  and  $h^{*}(x_{\perp})$  is non-decreasing in terms of  $\| x_{\perp}\| _2$  with  $h^{*}(0) = 0$ .
Assumption 4.2 adopts a simple linear reward model for ease of presentation. In this case, we estimate  $\theta^{*}$  by ridge regression, and (3.1) in Algorithm 1 becomes  $\widehat{\theta} = \mathrm{argmin}_{\theta}\sum_{i = 1}^{n_2}(\theta^\top x_i - y_i)^2 +\lambda \| \theta \| _2^2$  for a positive coefficient  $\lambda$ . Later in Section 3.3 and Appendix F, we extend our results beyond linear models to deep ReLU networks.
4.1 Conditional DM Learns Subspace Representation
Recall that Algorithm 1 has two outputs: generated population  $\widehat{P} (\cdot |\widehat{y} = a)$  and learned representation matrix  $V$ . Use notation  $\widehat{P}_a\coloneqq \widehat{P} (\cdot |\widehat{y} = a)$  (generated distribution) and  $P_{a}\coloneqq P(\cdot |\widehat{y} = a)$  (target distribution) for better clarity in result presentation. To assess the quality of subspace learning, we utilize two metrics defined as
$$
\angle (V, A) = \| V V ^ {\top} - A A ^ {\top} \| _ {F} ^ {2} \quad \text {a n d} \quad \mathbb {E} _ {x \sim \widehat {P} _ {a}} [ \| x _ {\perp} \| _ {2} ]. \tag {4.1}
$$
$\angle (V,A)$  is defined for matrices  $V,A$ , where  $A$  is the matrix encoding the ground truth subspace. Clearly,  $\angle (V,A)$  measures the difference in the column span of  $V$  and  $A$ , which is also known as the subspace angle.  $\mathbb{E}_{x\sim \widehat{P}_a}[\| x_\perp \| ]$  is defined as the expected  $l_{2}$  distance between  $x$  and the true subspace. Theorem 4.5 provides guarantees on this two metrics under following assumptions, proof and Interpretation of Theorem 4.5 are deferred to Appendix D.2.
To ease the presentation, we consider a Gaussian design on  $x$ , i.e. the latent  $z$  is Gaussian as stated in Assumption 4.4. Since our guarantee on  $\angle (V,A)$  holds under milder assumption than Gaussian, we also list the Assumption 4.3.
Assumption 4.3. The latent variable  $z$  follows distribution  $P_z$  with density  $p_z$ , such that there exists constants  $B, C_1, C_2$  verifying  $p_z(z) \leq (2\pi)^{-(d+1)/2} C_1 \exp \left(-C_2 \|z\|_2^2/2\right)$  whenever  $\|z\|_2 > B$ . And ground truth score is realizable:  $\nabla \log p_t(x \mid \widehat{y}) \in S$ .
Assumption 4.4. Further assume  $z \sim \mathsf{N}(0, \Sigma)$  with its covariance matrix  $\Sigma$  satisfying  $\lambda_{\min} I_d \preceq \Sigma \preceq \lambda_{\max} I_d$  for  $0 < \lambda_{\min} \leq \lambda_{\max} \leq 1$ .
Theorem 4.5 (Subspace Fidelity of Generated Data). Under Assumption 4.1, if Assumption 4.3 holds with  $c_{0}I_{d}\preceq \mathbb{E}_{z\sim P_{z}}[zz^{\top}]$ , then with high probability on data,
$$
\angle (V, A) = \widetilde {\mathcal {O}} \left(\frac {1}{c _ {0}} \sqrt {\frac {\mathcal {N} (\mathcal {S} , 1 / n _ {1}) D}{n _ {1}}}\right) \tag {4.2}
$$
with  $\mathcal{N}(\mathcal{S},1 / n_1)$  being the log covering number of function class  $\mathcal{S}$  as in (3.8). When Assumption 4.4 holds,  $\mathcal{N}(\mathcal{S},1 / n_1) = \mathcal{O}((d^2 +Dd)\log (Ddn_1))$  and thus  $\angle (V,A) = \widetilde{\mathcal{O}} (\frac{1}{\lambda_{\min}}\sqrt{\frac{(Dd^2 + D^2d)}{n_1}})$ . Further under Assumption 4.2, it holds that
$$
\mathbb {E} _ {x \sim \widehat {P} _ {a}} \left[ \| x _ {\perp} \| _ {2} \right] = \mathcal {O} \left(\sqrt {t _ {0} D} + \sqrt {\angle (V , A)} \cdot \sqrt {\frac {a ^ {2}}{\| \beta^ {*} \| _ {\Sigma}} + d}\right), \tag {4.3}
$$
where  $\beta^{*}$  is groundtruth parameter of linear model.
4.2 Provable Reward Improvement via Conditional Generation
Let  $y^{*}$  be a target reward value and  $P$  be a generated distribution. Define the suboptimality of  $P$  as
$$
\operatorname {S u b O p t} (P; y ^ {*}) = y ^ {*} - \mathbb {E} _ {x \sim P} [ f ^ {*} (x) ],
$$
which measures the gap between the expected reward of  $x \sim P$  and the target value  $y^{*}$ . In the language of bandit learning, this gap can also be viewed as a form of off-policy regret. Given a target value  $y^{*} = a$ , we want to derive guarantees for SubOpt  $(\widehat{P}_a; y^* = a)$ , recall  $\widehat{P}_a \coloneqq \widehat{P}(\cdot | \widehat{y} = a)$  denotes the generated distribution. In Theorem 4.6, we show SubOpt  $(\widehat{P}_a; y^* = a)$  comprises of three components: off-policy bandit regret which comes from the estimation error of  $\widehat{f}$ , on-support and off-support errors coming from approximating conditional distributions with diffusion.
Theorem 4.6 (Off-policy Regret of Generated Samples). Suppose Assumption 4.1, 4.2 and 4.4 hold. We choose  $\lambda = 1$ ,  $t_0 = \left((Dd^2 + D^2 d) / n_1\right)^{1/6}$  and  $\nu = 1/\sqrt{D}$ . With high probability, running Algorithm 1 with a target reward value  $a$  gives rise to
SubOpt  $(\widehat{P}_a; y^* = a)$
$$
\leq \underbrace {\sqrt {\operatorname {T r} \left(\widehat {\Sigma} _ {\lambda} ^ {- 1} \Sigma_ {P _ {a}}\right)} \cdot \mathcal {O} \left(\sqrt {\frac {d \log n _ {2}}{n _ {2}}}\right)} _ {\mathcal {E} _ {1}: \text {o f f - p o l i c y b a n d i t r e g r e t}} + \underbrace {\left| \mathbb {E} _ {P _ {a}} [ g ^ {*} (x _ {\parallel}) ] - \mathbb {E} _ {\widehat {P} _ {a}} [ g ^ {*} (x _ {\parallel}) ] \right|} _ {\mathcal {E} _ {2}: \text {o n - s u p p o r t d i f f u s i o n e r r o r}} + \underbrace {\mathbb {E} _ {\widehat {P} _ {a}} [ h ^ {*} (x _ {\perp}) ]} _ {\mathcal {E} _ {3}: \text {o f f - s u p p o r t d i f f u s i o n e r r o r}}, \tag {4.4}
$$
where  $\widehat{\Sigma}_{\lambda} := \frac{1}{n_2} (X^{\top}X + \lambda I)$  where  $X$  is the stack matrix of  $\mathcal{D}_{\mathrm{label}}$  and  $\Sigma_{P_a} = \mathbb{E}_{P_a}[xx^{\top}]$ .
Implications and Discussions: (1) Equation (4.4) decomposes the suboptimality gap into two separate parts of error: error from reward learning  $(\mathcal{E}_1)$  and error coming from diffusion  $(\mathcal{E}_2$  and  $\mathcal{E}_3$
(2) It is also worth mentioning that  $\mathcal{E}_2$  and  $\mathcal{E}_3$  depend on  $t_0$  and that by taking  $t_0 = ((Dd^2 + D^2 d) / n_1)^{1/6}$  one gets a good trade-off and small  $\mathcal{E}_2 + \mathcal{E}_3$ .
(3)  $\mathcal{E}_1$  suggests diffusion model is essentially doing representation learning, reducing  $D$  to smaller latent dimension  $d$ . It can be seen from  $\mathrm{Tr}(\widehat{\Sigma}_{\lambda}^{-1}\Sigma_{p_q}) \leq \mathcal{O}\left(\frac{a^2}{\|\beta^*\|_{\Sigma}} + d\right)$  when  $n_2 = \Omega\left(\frac{1}{\lambda_{min}}\right)$ .
(3) If we ignore the diffusion errors when  $n_1$  is large enough, the suboptimally gap resembles the suboptimality of off-policy bandit learning in feature subspace [22, Section 3.2], [32, 5]. It shows
the major source of error occurs when moving towards the target distributions and the error behaves similar to bandit.
(4) On-support diffusion error entangles with distribution shift in complicated ways. We show
$$
\mathcal {E} _ {2} = \left(\mathrm {D i s t r o S h i f t} (a) \cdot \left(d ^ {2} D + D ^ {2} d\right) ^ {1 / 6} n _ {1} ^ {- 1 / 6} \cdot a\right),
$$
where  $\text{DistroShift}(a)$  quantifies the distribution shift depending on different reward values. In the special case of the latent covariance matrix  $\Sigma$  is known, we can quantify the distribution shift as  $\text{DistroShift}(a) = \mathcal{O}(a \vee d)$ . We observe an interesting phase shift. When  $a < d$ , the training data have a sufficient coverage with respect to the generated distribution  $\widehat{P}_a$ . Therefore, the on-support diffusion error has a lenient linear dependence on  $a$ . However, when  $a > d$ , the data coverage is very poor and  $\mathcal{E}_2$  becomes quadratic in  $a$ , which quickly amplifies.
(5) When generated samples deviate away from the latent space, the reward may substantially degrade as determined by the nature of  $h$ .
To the authors' best knowledge, this is a first theoretical attempt to understand reward improvement of conditional diffusion. These results imply a potential connection between diffusion theory and off-policy bandit learning, which is interesting for more future research. See proofs in Appendix D.3.
4.3 Extension to Nonparametric Function Class
Our theoretical analysis, in its full generality, extends to using general nonparametric function approximation for both the reward and score functions. To keep our paper succinct, we refer to Appendix F and Theorem F.4 for details of our nonparametric theory for reward-conditioned generation. Informally, the regret of generated samples is bounded by
$$
\mathsf {S u b O p t} (\widehat {P} _ {a}; y ^ {*} = a) = \widetilde {\mathcal {O}} \left(\mathsf {D i s t r o S h i f t} (a) \cdot \left(n _ {2} ^ {- \frac {\alpha}{2 \alpha + d}} + n _ {1} ^ {- \frac {2}{3 (d + 6)}}\right)\right) + \mathbb {E} _ {\widehat {P} _ {a}} [ h ^ {*} (x _ {\perp}) ]
$$
with high probability. Additionally, the nonparametric generators is able to estimate the representation matrix  $A$  up to an error of  $\angle (V,A) = \widetilde{\mathcal{O}} (n_1^{-\frac{2}{d + 6}})$ . Here the score is assumed to be Lipschitz continuous and  $\alpha$  is the smoothness parameter of the reward function, and DistroShift(a) is a class-restricted distribution shift measure. Our results on nonparametric function approximation covers the use of deep ReLU networks as special cases.
We first perform the numerical simulation of Algorithm 1 following the setup in Assumption 4.1, 4.2 and 4.4. We choose  $d = 16$ ,  $D = 64$ ,  $g^{\star}(x) := 5\| x\|_{2}^{2}$ , and generate  $\beta^{\star}$  by uniformly sampling from the unit sphere. The latent variable  $z$  is generated from  $\mathsf{N}(0,\mathsf{I}_{\mathrm{d}})$ , which is then used to construct  $x = A z$  with some randomly generated orthonormal matrix  $A$ . We use the 1-dimensional version of the UNet [40] to approximate the score function. More details are deferred to Appendix I.
Figure 3 shows the average reward of the generated samples under different target reward values. We also plot the distribution shift and off-support deviation in terms of the 2-norm distance from the support. For small target reward values, the generation average reward almost scales linearly with the target value, which is consistent with the theory as the distribution shift remains small for these target values. The generation reward begins to decrease as we further increase the target reward value, and the reason is two fold. Firstly, the off-support deviation of the generated samples becomes large in this case, which prevents the generation reward from further going up. Secondly, the distribution shift increases rapidly as we further increase the target value, making the theoretical guarantee no longer valid. In Figure 4, we show the distribution of the rewards in the generated samples. As we increase the target reward values, the generation rewards become less concentrated and are shifted to the left of the target value, which is also due to the distribution shift and off-support deviation.
Figure 3: Quality of generated samples as target reward value increases. Left: Average reward of the generation; Middle: Distribution shift; Right: Off-support deviation. The errorbar is computed by 2 times the standard deviation over 5 runs.
Figure 4: Shifting reward distribution of the generated population.
5.2 Directed Text-to-Image Generation
Next, we empirically verify our theory through directed text-to-image generation. Instead of training a diffusion model from scratch, we use Stable Diffusion v1.5 [39], pre-trained on LAION dataset [42]. Stable Diffusion operates on the latent space of its Variational Auto-Encoder and can incorporate text conditions. We show that by training a reward model we can further guide the Stable Diffusion model to generate images of desired properties.
Ground-truth Reward Model. We start from an ImageNet [10] pre-trained ResNet-18 [17] model and replace the final prediction layer with a randomly initialized linear layer of scalar outputs. Then we use this model as the ground-truth reward model. To investigate the meaning of this randomly-generated reward model, we generate random samples and manually inspect the images with high rewards and low rewards. The ground-truth reward model seems to favor colorful and vivid natural scenes against monochrome and dull images; see Appendix I for sample images.
Labelled Dataset. We use the ground-truth reward model to compute a scalar output for each instance in the CIFAR-10 [24] training dataset and perturb the output by adding a Gaussian noise from  $\mathcal{N}(0, 0.01)$ . We use the images and the corresponding outputs as the training dataset.
Reward-network Training. To avoid adding additional input to the diffusion model and tuning the new parameters, we introduce a new network  $\mu_{\theta}$  and approximate  $p_t(y|x_t)$  by  $\mathsf{N}(\mu_{\theta}(\mathbf{x}_t),\sigma^2)$ . For simplicity, we set  $\sigma^2$  as a tunable hyperparameter. We share network parameters for different noise levels  $t$ , so our  $\mu_{\theta}$  has no additional input of  $t$ . We train  $\mu_{\theta}$  by minimizing the expected KL divergence between  $p_t(y|x_t)$  and  $\mathsf{N}(\mu_{\theta}(\mathbf{x}_t),\sigma^2)$ :
$$
\mathbb {E} _ {t} \mathbb {E} _ {x _ {t}} \left[ \mathrm {K L} \left(p _ {t} \left(y | x _ {t}\right) \mid \mathsf {N} \left(\mu_ {\theta} \left(x _ {t}\right), \sigma^ {2}\right)\right) \right] = \mathbb {E} _ {t} \mathbb {E} _ {\left(x _ {t}, y\right) \sim p _ {t}} \frac {\| y - \mu_ {\theta} \left(x _ {t}\right) \| _ {2} ^ {2}}{2 \sigma^ {2}} + \text {C o n s t a n t}.
$$
Equivalently, we train the reward model  $\mu_{\theta}$  to predict the noisy reward  $y$  from the noisy inputs  $x_{t}$ . Also, notice that the minimizers of the objective do not depend on the choice of  $\sigma^2$ .
Reward-network-based Directed Diffusion. To perform reward-directed conditional diffusion, observe that  $\nabla_{x}\log p_{t}(x|y) = \nabla_{x}\log p_{t}(x) + \nabla_{x}\log p_{t}(y|x)$ , and  $p_t(y|x)\propto \exp \left(-\frac{\|y - \mu_\theta(x)\|_2^2}{2\sigma^2}\right)$ . Therefore,
$$
\nabla_ {x} \log p _ {t} (y | x) = - 1 / \sigma^ {2} \cdot \nabla_ {x} \left[ \frac {1}{2} \| y - \mu_ {\theta} (x) \| _ {2} ^ {2} \right].
$$
In our implementation, we compute the gradient by back-propagation through  $\mu_{\theta}$  and incorporate this gradient guidance into each denoising step of the DDIM sampler [43] following [11] (equation (14)). We see that  $1 / \sigma^2$  corresponds to the weights of the gradient with respect to unconditioned score. In the sequel, we refer to  $1 / \sigma^2$  as the "guidance level", and  $y$  as the "target value".
Quantitative Results. We vary  $1 / \sigma^2$  in  $\{25, 50, 100, 200, 400\}$  and  $y$  in  $\{1, 2, 4, 8, 16\}$ . For each combination, we generate 100 images with the text prompt "A nice photo" and calculate the mean and the standard variation of the predicted rewards and the ground-truth rewards. The results are plotted in Figure 5. From the plot, we see similar effects of increasing the target value  $y$  at different guidance levels  $1 / \sigma^2$ . A larger target value puts more weight on the guidance signals  $\nabla_x \mu_\theta(x)$ , which successfully drives the generated images towards higher predicted rewards, but suffers more from the distribution-shift effects between the training distribution and the reward-conditioned distribution, which renders larger gaps between the predicted rewards and the ground-truth rewards. To optimally choose a target value, we must trade off between the two counteractive effects.
Figure 5: The predicted rewards and the ground-truth rewards of the generated images. At each guidance level, increasing the target  $y$  successfully directs the generation towards higher predicted rewards, but also increases the error induced by the distribution shift. The reported baseline is the expected ground-truth reward for undirected generations.
Qualitative Results. To qualitatively test the effects of the reward conditioning, we generate a set of images with increasing target values  $y$  under different text prompts and investigate the visual properties of the produced images. We isolate the effect of reward conditioning by fixing all the randomness during the generation processes, so the generated images have similar semantic layouts. After hyper-parameter tuning, we find that setting  $1 / \sigma^2 = 100$  and  $y \in \{2,4,6,8,10\}$  achieves good results across different text prompts and random seeds. We pick out typical examples and summarized the results in Figure 6, which demonstrates that as we increase the target value, the generated images become more colorful at the expense of degradations of the image qualities.
Reward improves as target value increases
Quality degrades when overly extrapolated
Figure 6: The effects of the reward-directed diffusion. Increasing the target value directs the images to be more colorful and vivid at the cost of degradation of the image qualities. Leftmost: without reward conditioning. Second-to-Last: target value  $y = 2, 4, 6, 8, 10$ . The guidance level  $1 / \sigma^2$  is fixed to 100. The text prompts are "A cat with a glass of water.", "An astronaut on the horseback".
In the paper, we study the problem of generating high-reward and high-quality samples using reward-directed conditional diffusion models, focusing on the semi-supervised setting where massive unlabeled data and limited labeled data are given. We provide theoretical results for subspace recovery and reward improvement, demonstrating the trade-off between the strength of the reward target and the distribution shift. Numerical results support our theory well.
Mengdi Wang acknowledges the support by NSF grants CPS-2312093, DMS-1953686, IIS-2107304, CMMI1653435, ONR grant 1006977 and C3.AI.
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=sP1fo2K9DFG.
[2] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993, 2021.
[4] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
[5] David Brandfonbrener, William Whitney, Rajesh Ranganath, and Joan Bruna. Offline contextual bandits with overparameterized models. In International Conference on Machine Learning, pages 1049-1058. PMLR, 2021.
[6] Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019.
[7] Minshuo Chen, Yu Bai, Jason D Lee, Tuo Zhao, Huan Wang, Caiming Xiong, and Richard Socher. Towards understanding hierarchical learning: Benefits of neural representations. Advances in Neural Information Processing Systems, 33:22134-22145, 2020.
[8] Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. arXiv preprint arXiv:2302.07194, 2023.
[9] Sitan Chen, Sinho Chewi, Jerry Li, Yuzhhi Li, Adil Salim, and Anru R Zhang. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. arXiv preprint arXiv:2209.11215, 2022.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[11] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[12] Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep q-learning. In Learning for Dynamics and Control, pages 486-489. PMLR, 2020.
[13] Sixue Gong, Vishnu Naresh Boddeti, and Anil K Jain. On the intrinsic dimensionality of image representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3987-3996, 2019.
[14] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras. Diffusion models as plug-and-play priors. arXiv preprint arXiv:2206.09012, 2022.
[15] László Györfi, Michael Köhler, Adam Krzyżak, and Harro Walk. A distribution-free theory of nonparametric regression, volume 1. Springer, 2002.
[16] Ulrich G Haussmann and Etienne Pardoux. Time reversal of diffusions. The Annals of Probability, pages 1188-1205, 1986.
[51] Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
[52] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674, 2011.
[53] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press, 2019.
[54] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
Index of Supplementary Materials
$\S A$  Preliminaries on Diffusion Models
$\S \mathbb{B}$  Limitations and Open Problems
$\S C$  Omitted Proof in Section 3
- §C.1 Proof of Proposition 3.1
$\S \mathbb{D}$  Omitted Proofs in Section 4
- §D.1 Parametric Conditional Score Matching Error
- §D.2 Proof of Theorem 4.5
- §D.3 Proof of Theorem 4.6
$\S E$  Supporting Lemmas and Proofs for  $\S D$
$\S F$  Extensions to Nonparametric Function Class
 §G Omitted Proofs in §F
$\S \mathrm{H}$  Proof for  $\S \mathrm{D}.1$
$\S \mathrm{I}$  Additional Experimental Results
A.1 Diffusion Models
We first provide a brief review of diffusion models and its training/sampling procedure. We consider diffusion in continuous time [23, 46], where diffusion is described as forward and backward SDEs.
Forward SDE and Score Matching. In the forward process, noise is added to original data progressively as an Ornstein-Ulthenbeck process for instance:
$$
\mathrm {d} X _ {t} = - \frac {1}{2} g (t) X _ {t} \mathrm {d} t + \sqrt {g (t)} \mathrm {d} W _ {t} \quad \text {f o r} g (t) > 0, \tag {A.1}
$$
where initial  $X_0 \sim P_{\mathrm{data}}$  and  $(W_t)_{t \geq 0}$  is a standard Wiener process, and  $g(t)$  is a nondecreasing weighting function. In practice, the forward process (A.1) terminates at a sufficiently large  $T > 0$  such that the corrupted  $X_T$  is close to the standard Gaussian  $\mathsf{N}(\mathbf{0}, I_D)$ . To enable data generation in future, the score  $\nabla \log p_t(\cdot)$  at  $t$  is the key to learn, here  $p_t$  denotes the marginal density of  $X_t$ . We often use an estimated score function  $\widehat{s}(\cdot, t)$  trained by minimizing a score matching loss.
Backward SDE for Generation. Diffusion models generate samples through a backward SDE (A.2) reversing the time in (A.1) [2, 16], i.e.,
$$
\mathrm {d} X _ {t} ^ {\leftarrow} = \left[ \frac {1}{2} g (T - t) X _ {t} ^ {\leftarrow} + g (T - t) \nabla \log p _ {T - t} \left(X _ {t} ^ {\leftarrow}\right) \right] \mathrm {d} t + \sqrt {g (T - t)} \mathrm {d} \bar {W} _ {t}, \tag {A.2}
$$
where  $\overline{W}_t$  is a reversed Wiener process. In practice, the backward process is initialized with  $\mathsf{N}(0,I_D)$  and the unknown conditional score  $\nabla \log p_t(\cdot)$  is replaced by an estimated counterpart  $\widehat{s} (\cdot ,t)$ .