SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhengyi Wang\*1,3, Cheng Lu\*1, Yikai Wang1, Fan Bao1,3, Chongxuan Li\*2, Hang Su1,4, Jun Zhu1 1,3,4
$^{1}$ Dept. of Comp. Sci. & Tech., BNrist Center, Tsinghua-Bosch Joint ML Center, Tsinghua University;  $^{2}$ Gaoling School of Artificial Intelligence, Renmin University of China, Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China  $^{3}$ ShengShu, Beijing, China;  $^{4}$ Pazhou Laboratory (Huangpu), Guangzhou, China {wang-zy21, bf19}@mails.tsinghua.edu.cn; lucheng.lc15@gmail.com; yikaiw@outlook.com; chongxuanli@ruc.edu.cn; suhangss@tsinghua.edu.cn dcszj@tsinghua.edu.cn
Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e.,  $512 \times 512$ ) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/proficdreamer/.
3D content and technologies enable us to visualize, comprehend, and interact with complex objects and environments that are reflective of our real-life experiences. Their pivotal role extends across a wide array of domains, encompassing architecture, animation, gaming, and the rapidly evolving fields of virtual and augmented reality. In spite of the extensive applications, the production of premium 3D content often remains a formidable task. It necessitates a significant investment of time and effort, even when undertaken by professional designers. This challenge has prompted the development of text-to-3D methods [19, 31, 34, 20, 4, 29, 55, 16]. By automating the generation of 3D content based on textual descriptions, these innovative methods present a promising way towards streamlining the 3D content creation process. Furthermore, they stand to make this process more accessible, potentially encouraging a significant paradigm shift in the aforementioned fields.
*Equal contribution; † Corresponding authors.
Michelangelo style statue of dog reading news on a cellphone.
A pineapple.
A chimpanzee dressed like Henry VIII king of England.
An elephant skull.
A model of a house in Tudor style.
A tarantula, highly detailed.
A snail on a leaf.
An astronaut is riding a horse.
(a) ProlificDreamer can generate meticulously detailed and photo-realistic 3D textured meshes.
A DSLR photo of a Space Shuttle.
A red fire hydrant spraying water.
A DSLR photo of a table with dim sum on it.
Inside of a smart home, realistic detailed photo, 4k.
A Matte painting of a castle made of cheesecake surrounded by a moat made of ice cream.
A DSLR photo of a hamburger inside a restaurant.
(b) ProlificDreamer can generate high rendering resolution (i.e.,  $512 \times 512$ ) and high-fidelity NeRF with rich structures and complex effects. Besides, the bottom results show that ProlificDreamer can generate complex scenes with  $360^{\circ}$  views because of our scene initialization (see Sec. 4.2).
A highly detailed sand castle.
(c) ProlificDreamer can generate diverse and semantically correct 3D scenes given the same text.
Figure 1: Text-to-3D samples generated by ProlificDreamer from scratch. Our base model is Stable Diffusion and we do not employ any other assistant model or user-provided shape guidance (see Table 1). See our accompanying videos in our project page for better visual quality.
A hotdog in a tutu skirt.
Diffusion models [46, 14, 49] have significantly advanced text-to-image synthesis [36, 41, 38, 1], particularly when trained on large-scale datasets [43]. Inspired by these developments, DreamFusion [34] employs a pretrained, large-scale text-to-image diffusion model for the generation of 3D content from text in the wild, circumventing the need for any 3D data. DreamFusion introduces the Score Distillation Sampling (SDS) algorithm to optimize a single 3D representation such that the image rendered from any view maintains a high likelihood as evaluated by the diffusion model, given the text. Despite its wide application [20, 4, 29, 55], empirical observations [34] indicate that SDS often suffers from over-saturation, over-smoothing, and low-diversity problems, which have yet to be thoroughly explained or adequately addressed. Additionally, orthogonal elements in the design space for text-to-3D, such as rendering resolution and distillation time schedule, have not been fully explored, suggesting a significant potential for further improvement. In this paper, we present a systematic study of all these elements to obtain elaborate 3D representations.
We first present Variational Score Distillation (VSD), which treats the corresponding 3D scene given a textual prompt as a random variable instead of a single point as in SDS [34]. VSD optimizes a distribution of 3D scenes such that the distribution induced on images rendered from all views aligns as closely as possible, in terms of KL divergence, with the one defined by the pretrained 2D diffusion model (see Sec. 3.1). Under this variational formulation, VSD naturally characterizes the phenomenon that multiple 3D scenes can potentially align with one prompt. To solve it efficiently, VSD adopts particle-based variational inference [23, 3, 9], and maintains a set of 3D parameters as particles to represent the 3D distribution. We derive a novel gradient-based update rule for the particles via the Wasserstein gradient flow (see Sec. 3.2) and guarantee that the particles will be samples from the desired distribution when the optimization converges (see Theorem 2). Our update requires estimating the score function of the distribution on diffused rendered images, which can be efficiently and effectively implemented by a low-rank adaptation (LoRA) [18, 40] of the pretrained diffusion model. The final algorithm alternatively updates the particles and score function.
We show that SDS is a special case of VSD, by using a single-point Dirac distribution as the variational distribution (see Sec. 3.3). This insight explains the restricted diversity and fidelity of the generated 3D scenes by SDS. Moreover, even with a single particle, VSD can learn a parametric score model, potentially offering superior generalization over SDS. We also empirically compare SDS and VSD in 2D space by using an identity rendering function that isolates other 3D factors. Similar to ancestral sampling from diffusion models, VSD is able to produce realistic samples using a normal CFG weight (i.e., 7.5). In contrast, SDS exhibits inferior results, sharing the same issues previously observed in text-to-3D, such as over-saturation and over-smoothing [34].
We further systematically study other elements orthogonal to the algorithm for text-to-3D and present a clear design space in Sec. 4. Specifically, we propose a high rendering resolution of  $512 \times 512$  during training and an annealed distilling time schedule to improve the visual quality. We also propose scene initialization, which is crucial for complex scene generation. Comprehensive ablations in Sec. 5 demonstrate the effectiveness of all the aforementioned elements particularly for VSD. Our overall approach can generate high-fidelity and diverse 3D results. We term it as *ProlificDreamer*<sup>2</sup>.
As shown in Fig. 1 and Sec. 5, ProlificDreamer can generate  $512 \times 512$  rendering resolution and high-fidelity Neural Radiance Fields (NeRF) with rich structure and complex effects (e.g., smoke and drops). Besides, for the first time, ProlificDreamer can successfully construct complex scenes with multiple objects in  $360^{\circ}$  views given the textual prompt. Further, initialized from the generated NeRF, ProlificDreamer can generate meticulously detailed and photo-realistic 3D textured meshes.
We present preliminaries on diffusion models, score distillation sampling, and 3D representations.
Diffusion models. A diffusion model [46, 14, 49] involves a forward process  $\{q_t\}_{t\in [0,1]}$  to gradually add noise to a data point  $\pmb{x}_0\sim q_0(\pmb{x}_0)$  and a reverse process  $\{p_t\}_{t\in [0,1]}$  to denoise/generate data. The forward process is defined by  $q_{t}(\pmb{x}_{t}|\pmb{x}_{0})\coloneqq \mathcal{N}(\alpha_{t}\pmb{x}_{0},\sigma_{t}^{2}\pmb {I})$  and  $q_{t}(\pmb{x}_{t}):= \int q_{t}(\pmb{x}_{t}|\pmb{x}_{0})q_{0}(\pmb{x}_{0})\mathrm{d}\pmb{x}_{0}$  where  $\alpha_{t},\sigma_{t} > 0$  are hyperparameters satisfying  $\alpha_0\approx 1,\sigma_0\approx 0,\alpha_1\approx 0,\sigma_1\approx 1$  ; and the reverse process is defined by denoising from  $p_1(x_1)\coloneqq \mathcal{N}(0,I)$  with a parameterized noise prediction
${}^{2}$  A prolific dreamer is someone who experiences vivid dreams quite regularly [50],which corresponds to the high-fidelity and diverse results of our method.
network  $\pmb{\epsilon}_{\phi}(\pmb{x}_t,t)$  to predict the noise added to a clean data  $\pmb{x}_0$  , which is trained by minimizing
$$
\mathcal {L} _ {\text {D i f f}} (\phi) := \mathbb {E} _ {\boldsymbol {x} _ {0} \sim q _ {0} \left(\boldsymbol {x} _ {0}\right), t \sim \mathcal {U} (0, 1), \epsilon \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I})} [ \omega (t) \| \epsilon_ {\phi} (\alpha_ {t} \boldsymbol {x} _ {0} + \sigma_ {t} \epsilon) - \epsilon \| _ {2} ^ {2} ], \tag {1}
$$
where  $\omega (t)$  is a time-dependent weighting function. After training, we have  $p_t\approx q_t$  and thus we can draw samples from  $p_0\approx q_0$ . Moreover, the noise prediction network can be used for approximating the score function of both  $q_{t}$  and  $p_t$  by  $\nabla_{\boldsymbol{x}_t}\log q_t(\boldsymbol {x}_t)\approx \nabla_{\boldsymbol{x}_t}\log p_t(\boldsymbol {x}_t)\approx -\epsilon_{\phi}(\boldsymbol {x}_t,t) / \sigma_t$ .
One of the most successful applications of diffusion models is text-to-image generation [41, 36, 38], where the noise prediction model  $\epsilon_{\phi}(\pmb{x}_t,t,y)$  is conditioned on a text prompt  $y$ . In practice, classifier-free guidance (CFG [15]) is a key technique for trading off the quality and diversity of the samples, which modifies the model by  $\hat{\epsilon}_{\phi}(\pmb{x}_t,t,y) \coloneqq (1 + s)\epsilon_{\phi}(\pmb{x}_t,t,y) - s\epsilon_{\phi}(\pmb{x}_t,t,\emptyset)$ , where  $\emptyset$  is a special "empty" text prompt representing the unconditional case, and  $s > 0$  is the guidance scale. A larger guidance scale usually improves the text-image alignment but reduces diversity.
Text-to-3D generation by score distillation sampling (SDS) [34]. SDS is an optimization method by distilling pretrained diffusion models, also known as Score Jacobian Chaining (SJC) [55]. It is widely used in text-to-3D generation [34, 55, 20, 29, 55, 4] with great promise. Given a pretrained text-to-image diffusion model  $p_t(\boldsymbol{x}_t|\boldsymbol{y})$  with the noise prediction network  $\epsilon_{\mathrm{pretrain}}(\boldsymbol{x}_t,t,\boldsymbol{y})$ , SDS optimizes a single 3D representation with parameter  $\theta \in \Theta$ , where  $\Theta$  is the space of  $\theta$  with the Euclidean metric. Given a camera parameter  $c$  with a distribution  $p(c)$  and a differentiable rendering mapping  $g(\cdot ,c):\Theta \to \mathbb{R}^d$ , denote  $y^{c}$  as the "view-dependent prompt" [34] (i.e., a text prompt with view information),  $q_{t}^{\theta}(\boldsymbol{x}_{t}|c)$  as the distribution at time  $t$  of the forward diffusion process starting from the rendered image  $g(\theta ,c)$  with the camera  $c$  and 3D parameter  $\theta$ . SDS optimizes the parameter  $\theta$  by solving
$$
\min  _ {\theta \in \Theta} \mathcal {L} _ {\mathrm {S D S}} (\theta) := \mathbb {E} _ {t, c} \left[ \left(\sigma_ {t} / \alpha_ {t}\right) \omega (t) D _ {\mathrm {K L}} \left(q _ {t} ^ {\theta} \left(\boldsymbol {x} _ {t} \mid c\right) \| p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right) \right], \tag {2}
$$
where  $t\sim \mathcal{U}(0.02,0.98)$ $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  , and  $\pmb {x}_t = \alpha_t\pmb {g}(\theta ,c) + \sigma_t\pmb {\epsilon}$  . Its gradient is approximated by
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {S D S}} (\theta) \approx \mathbb {E} _ {t, \epsilon , c} \left[ \omega (t) \left(\epsilon_ {\text {p r e t r a i n}} \left(\boldsymbol {x} _ {t}, t, y ^ {c}\right) - \epsilon\right) \frac {\partial \boldsymbol {g} (\theta , c)}{\partial \theta} \right]. \tag {3}
$$
Notwithstanding this progress, empirical observations [34] show that SDS often suffers from oversaturation, over-smoothing, and low-diversity issues, which have yet to be thoroughly explained or adequately addressed.
3D representations. We employ NeRF [30, 32] (Neural Radiance Fields) and textured mesh [45] as two popular and important types of 3D representations. In particular, NeRF represents 3D objects using a multilayer perceptron (MLP) that takes coordinates in a 3D space as input and outputs the corresponding color and density. Here,  $\theta$  corresponds to the parameters of the MLP. Given camera pose  $c$ , the rendering process  $g(\theta, c)$  is defined as casting rays from pixels and computing the weighted sum of the color of the sampling points along each ray to composite the color of each pixel. NeRF is flexible for optimization and is capable of representing extremely complex scenes. Textured mesh [45] represents the geometry of a 3D object with triangle meshes and the texture with color on the mesh surface. Here the 3D parameter  $\theta$  consists of the parameters to represent the coordinates of triangle meshes and parameters of the texture. The rendering process  $g(\theta, c)$  given camera pose  $c$  is defined by casting rays from pixels and computing the intersections between rays and mesh surfaces to obtain the color of each pixel. The textured mesh allows high-resolution and fast rendering with differentiable rasterization.
3 Variational Score Distillation
We now present Variational Score Distillation (VSD) (see Sec. 3.1) that learns to sample from a distribution of the 3D scenes. By using 3D parameter particles to represent the target 3D distribution, we derive a principled gradient-based update rule for the particles via the Wasserstein gradient flow (see Sec. 3.2). We further show that SDS is a special case of VSD and constructs an experiment in 2D space to study the optimization algorithm isolated from the 3D representations, explaining the practical issues of SDS both theoretically and empirically (see Sec. 3.3).
3.1 Sampling from 3D Distribution as Variational Inference
In principle, given a valid text prompt  $y$ , there exists a probabilistic distribution of all possible 3D representations. Under a 3D representation (e.g., NeRF) parameterized by  $\theta$ , such a distribution can
Figure 2: Overview of VSD. The 3D representation is differentiably rendered at a random pose  $c$ . The rendered image is sent to the pretrained diffusion and the score of the variational distribution (estimated by LoRA) to compute the gradient of VSD. LoRA is also updated on the rendered image.
be modeled as a probabilistic density  $\mu (\theta |y)$ . Denote  $q_0^\mu (\pmb {x}_0|c,y)$  as the (implicit) distribution of the rendered image  $\pmb{x}_0\coloneqq \pmb {g}(\theta ,c)$  given the camera  $c$  with the rendering function  $\pmb {g}(\cdot ,c)$ , and denote  $p_0(x_0|y^c)$  as the marginal distribution of  $t = 0$  defined by the pretrained text-to-image diffusion model with the view-dependent prompt  $y^{c}$ . To obtain 3D representations of high visual quality, we propose to optimize the distribution  $\mu$  to align the rendered images of its samples with the pretrained diffusion model in all views by solving
$$
\min  _ {\mu} D _ {\mathrm {K L}} \left(q _ {0} ^ {\mu} \left(\boldsymbol {x} _ {0} | c, y\right) \| p _ {0} \left(\boldsymbol {x} _ {0} \mid y ^ {c}\right)\right). \tag {4}
$$
This is a typical variational inference problem that uses the variational distribution  $q_0^\mu(\boldsymbol{x}_0|c, y)$  to approximate (distill) the target distribution  $p_0(\boldsymbol{x}_0|y^c)$ .
Directly solving problem (4) is hard because  $p_0$  is rather complex and the high-density regions of  $p_0$  may be extremely sparse in high dimension [47]. Inspired by the success of diffusion models [14, 49], we construct a series of optimization problems with different diffused distributions indexed by  $t$ . As  $t$  increases to  $T$ , the optimization problem becomes easier because the diffused distributions get closer to the standard Gaussian. We simultaneously solve an ensemble of these problems (termed as variational score distillation or VSD) as follows:
$$
\mu^ {*} := \underset {\mu} {\arg \min } \mathbb {E} _ {t, c} \left[ \left(\sigma_ {t} / \alpha_ {t}\right) \omega (t) D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) \| p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right) \right], \tag {5}
$$
where  $q_{t}^{\mu}(\pmb{x}_{t}|c,y) \coloneqq \int q_{0}^{\mu}(\pmb{x}_{0}|c,y)p_{t0}(\pmb{x}_{t}|\pmb{x}_{0})\mathrm{d}\pmb{x}_{0}$  and  $p_{t}(\pmb{x}_{t}|y^{c}) \coloneqq \int p_{0}(\pmb{x}_{0}|y^{c})p_{t0}(\pmb{x}_{t}|\pmb{x}_{0})\mathrm{d}\pmb{x}_{0}$  are the corresponding noisy distributions at time  $t$  with the Gaussian transition  $p_{t0}(\pmb{x}_{t}|\pmb{x}_{0}) = \mathcal{N}(\pmb{x}_{t}|\alpha_{t}\pmb{x}_{0},\sigma_{t}^{2}\pmb{I})$ , and  $\omega(t)$  is a time-dependent weighting function.
Compared with SDS that optimizes for the single point  $\theta$ , VSD optimizes for the whole distribution  $\mu$ , from which we sample  $\theta$ . Notably, we prove that introducing the additional KL-divergence for  $t > 0$  in VSD does not affect the global optimum of the original problem (4), as shown below.
Theorem 1 (Global optimum of VSD, proof in Appendix C.4.). For each  $t > 0$ , we have
$$
D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} | c, y\right) \parallel p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right) = 0 \Leftrightarrow q _ {0} ^ {\mu} \left(\boldsymbol {x} _ {0} \mid c, y\right) = p _ {0} \left(\boldsymbol {x} _ {0} \mid y ^ {c}\right). \tag {6}
$$
3.2 Update Rule for Variational Score Distillation
To solve problem (5), a direct way can be to train another parameterized generative model for  $\mu$ , but it may bring much computation cost and optimization complexity. Inspired by previous particle-based variational inference [23, 3, 9] methods, we maintain  $n$  3D parameters $^3$ $\{\theta\}_{i=1}^n$  as particles and derive a novel update rule for them. Intuitively, we use  $\{\theta\}_{i=1}^n$  to "represent" the current distribution  $\mu$ , and  $\theta^{(i)}$  will be samples from the optimal distribution  $\mu^*$  if the optimization converges. Such optimization can be realized by simulating an ODE w.r.t.  $\theta$ , as shown in the following theorem.
Theorem 2 (Wasserstein gradient flow of VSD, proof in Appendix C). Starting from an initial distribution  $\mu_0$ , denote the Wasserstein gradient flow minimizing problem (5) in the distribution (function) space at each time  $\tau \geq 0$  as  $\{\mu_{\tau}\}_{\tau \geq 0}$  with  $\mu_{\infty} = \mu^{*}$ . Then we can sample  $\theta_{\tau}$  from  $\mu_{\tau}$  by
<sup>3</sup>We optimize up to  $n = 4$  particles due to the computation resource limit. See details in Appendix D.3.
(a) SDS [34] (CFG = 7.5)
(b) SDS [34] (CFG = 100)
(c) Ancestral sampling [27] (CFG = 7.5)
(d) VSD (CFG = 7.5, ours)
Figure 3: Samples of different methods in 2D space. Similarly to ancestral sampling, VSD generates realistic images with a common CFG weight of 7.5 and outperforms SDS significantly. The prompts from left to right are hamburger, horse, and a monster truck, respectively. See details in Appendix G.
firstly sampling  $\theta_0\sim \mu_0(\theta_0|y)$  and then simulating the following ODE:
$$
\frac {\mathrm {d} \theta_ {\tau}}{\mathrm {d} \tau} = - \mathbb {E} _ {t, \epsilon , c} \left[ \omega (t) \left(\underbrace {- \sigma_ {t} \nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} (\boldsymbol {x} _ {t} | y ^ {c})} _ {\text {s c o r e o f n o i s y r e a l i m a g e s}} - \underbrace {\left(- \sigma_ {t} \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu_ {\tau}} (\boldsymbol {x} _ {t} | c , y)\right)} _ {\text {s c o r e o f n o i s y r e n d e r e d i m a g e s}}\right) \frac {\partial \boldsymbol {g} (\theta_ {\tau} , c)}{\partial \theta_ {\tau}} \right], \tag {7}
$$
where  $q_{t}^{\mu_{\tau}}$  is the corresponding noisy distribution at diffusion time  $t$  w.r.t.  $\mu_{\tau}$  at ODE time  $\tau$ .
According to Theorem 2, we can simulate the ODE in Eq. (7) for a large enough  $\tau$  to approximately sample from the desired distribution  $\mu^{*}$ . The ODE involves the score function of noisy real images and that of noisy rendered images at each time $^{4}$ $\tau$ . The score function of noisy real images  $-\sigma_{t}\nabla_{\boldsymbol{x}_{t}}\log p_{t}(\boldsymbol{x}_{t}|y^{c})$  can be approximated by the pretrained diffusion model  $\epsilon_{\mathrm{pretrain}}(\boldsymbol{x}_t,t,y^c)$ . The score function of noisy rendered images  $-\sigma_{t}\nabla_{\boldsymbol{x}_{t}}\log q_{t}^{\mu_{\tau}}(\boldsymbol{x}_{t}|c,y)$  is estimated by another noise prediction network  $\epsilon_{\phi}(\boldsymbol{x}_{t},t,c,y)$ , which is trained on the rendered images by  $\{\theta^{(i)}\}_{i = 1}^{n}$  with the standard diffusion objective (see Eq. (1)):
$$
\min  _ {\phi} \sum_ {i = 1} ^ {n} \mathbb {E} _ {t \sim \mathcal {U} (0, 1), \epsilon \sim \mathcal {N} (\boldsymbol {0}, \boldsymbol {I}), c \sim p (c)} \left[ \| \epsilon_ {\phi} \left(\alpha_ {t} \boldsymbol {g} \left(\theta^ {(i)}, c\right) + \sigma_ {t} \epsilon , t, c, y\right) - \epsilon \| _ {2} ^ {2} \right]. \tag {8}
$$
In practice, we parameterize  $\epsilon_{\phi}$  by either a small U-Net [39] or a LoRA (Low-rank adaptation [18, 40]) of the pretrained model  $\epsilon_{\mathrm{pretrain}}(x_t,t,y^c)$ , and add additional camera parameter  $c$  to the condition embeddings in the network. In most cases, we find that using LoRA can greatly improve the fidelity of the obtained samples (e.g., see results in Fig. 1). We believe that it is because LoRA is designed for efficient few-shot fine-tuning and can leverage the prior information in  $\epsilon_{\mathrm{pretrain}}$  (the information of both images and text corresponding to  $y$ ).
Note that at each ODE time  $\tau$ , we need to ensure  $\epsilon_{\phi}$  matches the current distribution  $q_{t}^{\mu_{\tau}}$ . Thus, we optimize  $\epsilon_{\phi}$  and  $\theta^{(i)}$  alternately, and each particle  $\theta^{(i)}$  is updated by  $\theta^{(i)} \gets \theta^{(i)} - \eta \nabla_{\theta} \mathcal{L}_{\mathrm{VSD}}(\theta^{(i)})$ , where  $\eta > 0$  is the step size (learning rate). According to Theorem 2, the corresponding gradient is
$$
\nabla_ {\theta} \mathcal {L} _ {\mathrm {V S D}} (\theta) \triangleq \mathbb {E} _ {t, \epsilon , c} \left[ \omega (t) \left(\epsilon_ {\text {p r e t r a i n}} \left(\boldsymbol {x} _ {t}, t, y ^ {c}\right) - \epsilon_ {\phi} \left(\boldsymbol {x} _ {t}, t, c, y\right)\right) \frac {\partial \boldsymbol {g} (\theta , c)}{\partial \theta} \right], \tag {9}
$$
where  $\pmb{x}_t = \alpha_t\pmb{g}(\theta, c) + \sigma_t\pmb{\epsilon}$ . We show the approach of VSD in Fig. 3 (see pseudo code in Appendix E).
3.3 Comparison with SDS