SlowGuess/NeurIPS_2023_OCR huggingface dataset:
starts with some  $(n)$  particles  $\{\theta_0^{(i)}\}_{i = 1}^n$  sampled from an initial distribution  $\mu_0\in \mathcal{P}(\Theta)$  and update these particles with a vector field  $v_{\tau}(\cdot)$  on  $\Theta$  by  $\mathrm{d}\theta_{\tau}^{(i)} = v_{\tau}(\theta_{\tau}^{(i)})\mathrm{d}\tau$  at each time  $\tau \geq 0$ , such that the distribution  $\mu_{\tau}$  of  $\{\theta_{\tau}^{(i)}\}_{i = 0}^{n}$  converges to  $\mu^{*}$  as  $\tau \to \infty$  and  $n\to \infty$ . Typically, the measure  $\mu_{\tau}$  follows the "steepest descending curves" of a functional on  $\mathbb{W}_2(\Theta)$ , which is known as the Wasserstein gradient flow. A common setting is to define the functional of  $\mu_{\tau}$  as the KL-divergence  $D_{\mathrm{KL}}(\mu_{\tau}\parallel \mu^{*})$ , and then  $\mu_{\tau}$  follows  $\partial_{\tau}\mu_{\tau} = -\nabla_{\theta}\cdot (\mu_{\tau}\nabla_{\theta}\log \frac{\mu^{*}}{\mu_{\tau}})$ , and the corresponding vector field is  $v_{\tau}(\theta_{\tau}) = \nabla_{\theta}\log \frac{\mu^{*}}{\mu_{\tau}}$ . However, directly computing the vector field is intractable because it is non-trivial to compute the time-dependent score function  $\nabla_{\theta}\log \mu_{\tau}(\theta)$ . To tackle this issue, traditional ParVI methods either restrict the functional gradient within RKHS and leverage analytical kernels to approximate the vector field [23, 3, 22], or learn a neural network to estimate the vector field [8]; but all of these methods are hard to scale up to high-dimensional data such as images.
C.2 VSD as Particle-based Variational Inference
In this section, we formally propose the theoretical guarantee of VSD.
Denote  $\mathbb{W}_2(\Theta)$  as the 2-Wasserstein space [53] of the probability distributions defined on the parameter space  $\Theta$ . For any distribution  $\mu \in \mathbb{W}_2(\Theta)$  with a corresponding random variable  $\theta \sim \mu(\theta | y)$ , denote the implicit distribution for  $\pmb{x}_0 := \pmb{g}(\theta, c)$  as  $q_0^\mu(\pmb{x}_0 | c, y)$ , and let  $q_t^\mu(\pmb{x}_t | c, y) := \int q_0^\mu(\pmb{x}_0 | c, y) q_{t0}(\pmb{x}_t | \pmb{x}_0) \mathrm{d}\pmb{x}_0$  be the marginal distribution of the random variable  $\pmb{x}_t$  during the diffusion process. We want to optimize the distribution  $\mu(\theta | y)$  in the function space  $\mathbb{W}_2(\Theta)$  by distilling a pretrained diffusion model  $p_t(\pmb{x}_t | y^c)$  via minimizing the following functional of  $\mu$ :
$$
\min  _ {\mu \in \mathbb {W} _ {2} (\Theta)} \mathcal {E} [ \mu ] := \mathbb {E} _ {t, \epsilon , c} \left[ \frac {\sigma_ {t}}{\alpha_ {t}} \omega (t) D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) \| p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right) \right]. \tag {10}
$$
Compared with the vanilla SDS in Eq. (2) which optimizes the parameter  $\theta$  in the parameter space  $\Theta$ , here we aim to optimize the distribution (measure)  $\mu(\theta | y)$  in the function space  $\mathbb{W}_2(\Theta)$ . Inspired by previous ParVI techniques [23, 3, 56], we firstly derive the gradient flow minimizing  $\mathcal{E}[\mu]$  in  $\mathbb{W}_2(\Theta)$  and the update rule of each particle  $\theta \sim \mu(\theta)$ , as shown in the following theorem.
Theorem 3 (Wasserstein gradient flow of score distillation). Starting from an initial distribution  $\mu_0(\theta |y)$ , the gradient flow  $\{\mu_{\tau}\}_{\tau \geq 0}$  minimizing  $\mathcal{E}[\mu_{\tau}]$  on  $\mathbb{W}_2(\Theta)$  at each time  $\tau \geq 0$  satisfies
$$
\left. \frac {\partial \mu_ {\tau} (\theta | y)}{\partial \tau} = - \nabla_ {\theta} \cdot \left[ \mu_ {\tau} (\theta | y) \mathbb {E} _ {t, \epsilon , c} \left[ \sigma_ {t} \omega (t) \left(\nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu_ {\tau}} \left(\boldsymbol {x} _ {t} \mid c, y\right)\right) \frac {\partial \boldsymbol {g} (\theta , c)}{\partial \theta} \right] \right], \right. \tag {11}
$$
and the corresponding update rule for each  $\theta_{\tau} \sim \mu_{\tau}$  satisfies
$$
\frac {\mathrm {d} \theta_ {\tau}}{\mathrm {d} \tau} = - \mathbb {E} _ {t, \boldsymbol {\epsilon}, c} \left[ \omega (t) \left(\underbrace {- \sigma_ {t} \nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)} _ {\approx \boldsymbol {\epsilon} _ {\text {p r e t r a i n}} \left(\boldsymbol {x} _ {t}, t, y ^ {c}\right)} - \underbrace {\left(- \sigma_ {t} \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu_ {\tau}} \left(\boldsymbol {x} _ {t} \mid c , y\right)\right)} _ {\approx \boldsymbol {\epsilon} _ {\phi} \left(\boldsymbol {x} _ {t}, t, c, y\right)}\right) \frac {\partial \boldsymbol {g} \left(\theta_ {\tau} , c\right)}{\partial \theta_ {\tau}} \right]. \tag {12}
$$
Theorem 3 shows that by letting the random variable  $\theta_{\tau} \sim \mu_{\tau}(\theta_{\tau}|y)$  move across the ODE trajectory in Eq. (12), its underlying distribution  $\mu_{\tau}$  will move by the direction of the steepest descent that minimizes  $\mathcal{E}[\mu]$ . Therefore, to obtain samples (in  $\Theta$ ) from  $\mu^{*} = \arg \min_{\mu} \mathcal{E}[\mu]$ , we can simulate the ODE in Eq. (12) by estimating two score functions  $\nabla_{\boldsymbol{x}_t} \log p_t(\boldsymbol{x}_t|y^c)$  and  $\nabla_{\boldsymbol{x}_t} \log q_t^{\mu_\tau}(\boldsymbol{x}_t|c,y)$  at each ODE time  $\tau$ , which corresponds to the VSD objective in Eq. (9).
C.3 Discussions of SDS / SJC and VSD
There are three main differences between our proposed VSD and previous SDS [34] / SJC [55]: (1) VSD can optimize multiple  $(n\geq 1)$  3D objects, while SDS / SJC only optimizes a single  $(n = 1)$  3D object. (2) VSD can generate high-fidelity results with the common (e.g., 7.5) CFG (for both  $n = 1$  and  $n > 1$ ), while SDS / SJC needs extremely large (e.g. 100) CFG. (3) SDS / SJC is a special case of VSD by using a single-point Dirac distribution  $\mu (\theta |y)\approx \delta (\theta -\theta^{(1)})$  as the variational distribution.
SDS / SJC as a special case of VSD. Here we explain SDS / SJC under our variational inference framework. If we directly approximate  $\mu(\theta | y) \approx \delta(\theta - \theta^{(1)})$  by its empirical distribution with the single sample  $\theta^{(1)}$ , then we have  $q_{t}^{\mu}(\pmb{x}_{t} | c, y) \approx \mathcal{N}(\pmb{x}_{t} | \alpha_{t} \pmb{g}(\theta^{(1)}, c), \sigma_{t}^{2} \pmb{I})$  and thus  $-\sigma_{t} \nabla_{\pmb{x}_{t}} \log q_{t}^{\mu}(\pmb{x}_{t} | c, y) \approx (\pmb{x}_{t} - \alpha_{t} \pmb{g}(\theta^{(1)}, c)) / \sigma_{t} = \epsilon$ , which recovers the vanilla SDS / SJC. Therefore, SDS / SJC is a special case of VSD for the underlying distribution  $\mu(\theta | y)$  by the empirical distribution with a single point. Such an approximation has no generalization ability for  $\theta \neq \theta^{(1)}$ , and thus the updating direction by the score function  $-\sigma_{t} \nabla_{\pmb{x}_{t}} \log q_{t}^{\mu}(\pmb{x}_{t} | c, y)$  (or equivalently, the Gaussian noise  $\epsilon$ ) may be rather bad at low-density regions, resulting poor samples for the final  $\theta^{(1)}$ . Instead, VSD regards  $\theta^{(1)}$  as samples from the underlying distribution  $\mu(\theta | y)$  and trains a neural network  $\epsilon_{\phi}$  to approximate the corresponding score functions, thus can leverage the generalization ability of neural networks for better approximating the underlying distribution  $\mu(\theta | y)$ . Moreover, by using LoRA, VSD can additionally exploit the text prompt  $y$  in the estimation  $\epsilon_{\phi}(\pmb{x}_{t}, t, c, y)$ , while the Gaussian noise  $\epsilon$  used in SDS cannot leverage the information from  $y$ .
SDS / SJC as mode-seeking, VSD as sampling. VSD aims to sample  $\theta$  from the optimal  $\mu^{*}$ , while SDS / SJC aims to find the optimal  $\theta^{*}$  minimizing the objective in Eq. (2). As the global optimum  $\theta^{*}$  in SDS / SJC is the mode of Eq. (2), SDS / SJC is also known as performing mode-seeking [34]. To demonstrate the differences between sampling and mode-seeking, we consider a special case of the rendering function  $g(\theta)$  to decouple the optimization algorithm and the 3D representation. In particular, we set  $g(\theta, c) \equiv \theta$  for any  $c$  and  $\theta \in \mathbb{R}^d$ , then the rendered image  $\pmb{x} = \pmb{g}(\theta, c) = \theta$  is the same 2D image as  $\theta$ . In such a case, we have  $q_0^\mu = \mu$ , and it is easy to prove that  $\mu^{*} = p_{0}$  (according to Theorem 1). For VSD, sampling  $\theta$  from  $\mu^{*}$  is corresponding to the traditional ancestral sampling [27] from  $p_0$ ; while for SDS / SJC, we have  $\nabla_{\theta} \mathcal{L}_{\mathrm{SDS}}(\theta) = \mathbb{E}_{t,\epsilon}[\omega(t) \epsilon_{\mathrm{pretrain}}(\pmb{x}_t, t, y^c)] \approx \mathbb{E}_{t,\epsilon}[-\sigma_t \omega(t) \nabla_{\pmb{x}_t} \log p_t(\pmb{x}_t | y^c)]$ , and thus the optimal  $\theta^{*}$  is the mode of the "averaged likelihood" of  $p_t$  for all  $t$ . However, it is common that the mode of deep generative models may have poor sample quality [33]. We show in Fig. 3 that under the same CFG (7.5), both VSD and ancestral sampling can generate good samples but the sample quality of SDS is quite poor.
SDS / SJC requires large CFG, while VSD is friendly to CFG. As VSD aims to sample  $\theta$  from the optimal  $\mu^{*}$  defined by the pretrained model  $\epsilon_{\mathrm{pretrain}}$ , the effects by tuning the CFG in  $\epsilon_{\mathrm{pretrain}}$  for
3D samples  $\theta$  by VSD are quite similar to the effects for the 2D samples by the traditional ancestral sampling [14, 27]. Therefore, VSD can tune CFG as flexibly as the classic text-to-image methods, and we use the same setting of CFG (e.g. 7.5) as the common text-to-image generation task for the best performance. To the best of our knowledge, this for the first time addresses the problem in previous SDS [34, 20, 4, 29] that it usually requires a large CFG (i.e., 100). Specifically, SDS (Eq. (3)) uses  $(\epsilon_{\mathrm{pretrain}}(\boldsymbol{x}_t,t,y^c) - \epsilon)$  while VSD (Eq. (9)) uses  $(\epsilon_{\mathrm{pretrain}}(\boldsymbol{x}_t,t,y^c) - \epsilon_\phi (\boldsymbol{x}_t,t,c,y))$ . For example, for the 2D special case of  $\pmb {g}(\theta ,c)\equiv \theta$ , we have  $\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\theta) = \mathbb{E}_{t,\epsilon}[\omega (t)\epsilon_{\mathrm{pretrain}}(\boldsymbol{x}_t,t,y^c)]$  and  $\nabla_{\theta}\mathcal{L}_{\mathrm{VSD}}(\theta) = \mathbb{E}_{t,\epsilon}[\omega (t)(\epsilon_{\mathrm{pretrain}}(\boldsymbol{x}_t,t,y^c) - \epsilon_\phi (\boldsymbol{x}_t,t,c,y))]$ . Intuitively, to obtain highly detailed samples, the updating direction for  $\theta$  needs to be "fine" and "sharp". As SDS only depends on  $\epsilon_{\mathrm{pretrain}}$ , it needs a large CFG (= 100) to make sure  $\epsilon_{\mathrm{pretrain}}$  to be "sharp" enough; however, large CFG, in turn, reduces the diversity of the results and also hurts the quality. Instead, VSD leverages an additional score function  $\epsilon_{\phi}(\boldsymbol{x}_t,t,c,y)$  to give a more elaborate direction than the zero-mean Gaussian noise, and the updating direction can be rather "fine" and "sharp" due to the difference between  $\epsilon_{\mathrm{pretrain}}$  and  $\epsilon_{\phi}$ . We empirically find that VSD can obtain much better quality than SDS in both 2D and 3D generation.
C.4 Proof of Main Theorem
C.4.1 Proof of Theorem 1
Proof of Theorem 1. Denote  $\pmb{x}_0^q$  as the random variable following  $q_0^\mu (\pmb{x}_0|c,y)$  and  $\pmb{x}_t^q$  as the random variable following  $q_t^\mu (\pmb{x}_t|c,y)$ . By the definition of  $q_{t}^{\mu}$ , we have
$$
\boldsymbol {x} _ {t} ^ {q} = \alpha_ {t} \boldsymbol {x} _ {0} ^ {q} + \sigma_ {t} \boldsymbol {\epsilon}, \tag {13}
$$
where  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ . Therefore, the characteristic functions of  $q_{t}^{\mu}$  and  $q_0^\mu$  satisfy
$$
\varphi_ {q _ {t} ^ {\mu}} (s) = \varphi_ {q _ {0} ^ {\mu}} (\alpha_ {t} s) \cdot \varphi_ {\mathcal {N} (\mathbf {0}, \mathbf {I})} (\sigma_ {t} s) = e ^ {- \frac {\sigma_ {t} ^ {2} s ^ {2}}{2}} \varphi_ {q _ {0} ^ {\mu}} (\alpha_ {t} s). \tag {14}
$$
Similarly, the characteristic functions of  $p_t$  and  $p_0$  satisfy
$$
\varphi_ {p _ {t}} (s) = e ^ {- \frac {\sigma_ {t} ^ {2} s ^ {2}}{2}} \varphi_ {p _ {0}} (\alpha_ {t} s). \tag {15}
$$
$$
D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) \| p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right) = 0 \Leftrightarrow q _ {t} ^ {\mu} = p _ {t} \Leftrightarrow \varphi_ {q _ {t} ^ {\mu}} = \varphi_ {p _ {t}} \Leftrightarrow \varphi_ {q _ {0} ^ {\mu}} = \varphi_ {p _ {0}} \Leftrightarrow q _ {0} ^ {\mu} = p _ {0} \tag {16}
$$
C.4.2 Proof of Theorem 3
As the gradient flow minimizing  $\mathcal{E}[\mu ]$  on  $\mathbb{W}_2(\Theta)$  satisfies
$$
\frac {\partial \mu_ {\tau}}{\partial \tau} = - \nabla_ {\mathbb {W} _ {2}} \mathcal {E} [ \mu ] = \nabla_ {\theta} \cdot \left(\mu_ {\tau} \nabla_ {\theta} \frac {\delta \mathcal {E} [ \mu_ {\tau} ]}{\delta \mu_ {\tau}}\right), \tag {17}
$$
thus we only need to compute the first variation  $\frac{\delta\mathcal{E}[\mu]}{\delta\mu}$ . We propose the following lemmas for computing  $\frac{\delta\mathcal{E}[\mu]}{\delta\mu}$ .
Lemma 1. For  $p,q\in \mathbb{W}_2(\mathbb{R}^d)$  , for any  $\pmb {x}\in \mathbb{R}^d$
$$
\left(\frac {\delta D _ {\mathrm {K L}} (q \| p)}{\delta q}\right) [ \boldsymbol {x} ] = \log q (\boldsymbol {x}) - \log p (\boldsymbol {x}) + 1 \tag {18}
$$
Proof. This is a classic conclusion in particle-based variational inference (e.g., see Sec.3.2. in [3]).  $\square$
Lemma 2. For a fixed  $c$  and  $\pmb{x}_t$ ,
$$
\frac {\delta q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c , y\right)}{\delta \mu} [ \theta ] = q _ {t 0} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {0}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t} \mid \alpha_ {t} \boldsymbol {x} _ {0}, \sigma_ {t} ^ {2} \boldsymbol {I}\right), \tag {19}
$$
where  $\pmb{x}_0 = \pmb{g}(\theta, c)$ .
Proof of Lemma 2. By the definition of  $q_{t}^{\mu}$ , we have
$$
q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) = \mathbb {E} _ {q _ {0} ^ {\mu} \left(\boldsymbol {x} _ {0} \mid c, y\right)} \left[ q _ {t 0} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {0}\right) \right] = \mathbb {E} _ {\mu (\theta \mid y)} \left[ q _ {t 0} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {g} (\theta , c)\right) \right], \tag {20}
$$
$$
\frac {\delta q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c , y\right)}{\delta \mu} [ \theta ] = q _ {t 0} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {g} (\theta , c)\right) = \mathcal {N} \left(\boldsymbol {x} _ {t} \mid \alpha_ {t} \boldsymbol {g} (\theta , c), \sigma_ {t} ^ {2} \boldsymbol {I}\right) \tag {21}
$$
$$
\frac {\delta D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c , y\right) \parallel p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right)}{\delta \mu} [ \theta ] = \mathbb {E} _ {\epsilon} \left[ \log q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) - \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right) + 1 \right], \tag {22}
$$
where  $\pmb{x}_t = \alpha_t\pmb{g}(\theta, c) + \sigma_t\pmb{\epsilon}, \epsilon \sim \mathcal{N}(\pmb{\epsilon}|\pmb{0}, \pmb{I})$ .
Proof of Lemma 3. By the chain rule of functional derivative, according to Lemma 1 and Lemma 2, we have
$$
\begin{array}{l} \frac {\delta D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c , y\right) \| p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right)}{\delta \mu} [ \theta ] = \int \frac {\delta D _ {\mathrm {K L}} \left(q _ {t} ^ {\mu} \| p _ {t}\right)}{\delta q _ {t} ^ {\mu}} [ \boldsymbol {x} _ {t} ] \cdot \frac {\delta q _ {t} ^ {\mu} (\boldsymbol {x} _ {t} | c , y)}{\delta \mu} [ \theta ] \mathrm {d} \boldsymbol {x} _ {t} (23) \\ = \int \left(\log q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) - \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right) + 1\right) q _ {t 0} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {0}\right) \mathrm {d} \boldsymbol {x} _ {t} (24) \\ = \mathbb {E} _ {q _ {t 0} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {0})} \left[ \log q _ {t} ^ {\mu} (\boldsymbol {x} _ {t} | c, y) - \log p _ {t} (\boldsymbol {x} _ {t} | y ^ {c}) + 1 \right] (25) \\ = \mathbb {E} _ {\epsilon} \left[ \log q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} \mid c, y\right) - \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right) + 1 \right], (26) \\ \end{array}
$$
where  $\pmb {\epsilon}\sim \mathcal{N}(\pmb {\epsilon}|\pmb {0},\pmb {I}),\pmb{x}_0 = \pmb {g}(\theta ,c),\pmb{x}_t = \alpha_t\pmb{x}_0 + \sigma_t\pmb {\epsilon}.$
Below we provide the proof of Theorem 3.
Proof of Theorem 3. According to Lemma 3, we have
$$
\frac {\delta \mathcal {E} [ \mu ]}{\delta \mu} [ \theta ] = \mathbb {E} _ {t, \epsilon , c} \left[ \frac {\sigma_ {t}}{\alpha_ {t}} \omega (t) \left(\log q _ {t} ^ {\mu} \left(\boldsymbol {x} _ {t} | c, y\right) - \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right) + 1\right) \right], \tag {27}
$$
where  $\pmb{x}_t = \alpha_t\pmb{g}(\theta, c) + \sigma_t\pmb{\epsilon}$ . So
$$
\begin{array}{l} \nabla_ {\theta} \frac {\delta \mathcal {E} [ \mu ]}{\delta \mu} [ \theta ] = \mathbb {E} _ {t, \boldsymbol {\epsilon}, c} \left[ \frac {\sigma_ {t}}{\alpha_ {t}} \omega (t) \left(\nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu} (\boldsymbol {x} _ {t} | c, y) - \nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} (\boldsymbol {x} _ {t} | y ^ {c})\right) \frac {\partial \boldsymbol {x} _ {t}}{\partial \theta} \right] (28) \\ = \mathbb {E} _ {t, \epsilon , c} \left[ \sigma_ {t} \omega (t) \left(\nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu} (\boldsymbol {x} _ {t} | c, y) - \nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} (\boldsymbol {x} _ {t} | y ^ {c})\right) \frac {\partial \boldsymbol {g} (\theta , c)}{\partial \theta} \right]. (29) \\ \end{array}
$$
Thus, the measure  $\mu_{\tau}$  at step  $\tau$  during the Wasserstein gradient flow minimizing  $\mathcal{E}[\mu]$  on  $\mathbb{W}_2(\Theta)$  satisfies
$$
\begin{array}{l} \frac {\partial \mu_ {\tau}}{\partial \tau} = - \nabla_ {\mathbb {W} _ {2}} \mathcal {E} [ \mu ] (30) \\ = \nabla_ {\theta} \cdot \left(\mu_ {\tau} \nabla_ {\theta} \frac {\delta \mathcal {E} [ \mu_ {\tau} ]}{\delta \mu_ {\tau}}\right) (31) \\ = \nabla_ {\theta} \cdot \left[ \mu_ {\tau} (\theta | y) \mathbb {E} _ {t, \boldsymbol {\epsilon}, c} \left[ \sigma_ {t} \omega (t) \left(\nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu_ {\tau}} \left(\boldsymbol {x} _ {t} \mid c, y\right) - \nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right)\right) \frac {\partial \boldsymbol {g} (\theta , c)}{\partial \theta} \right] \right]. (32) \\ \end{array}
$$
By the definition of Fokker-Planck equation, the corresponding process of each particle  $\theta_{\tau}$  at time  $\tau$  satisfies
$$
\frac {\mathrm {d} \theta_ {\tau}}{\mathrm {d} \tau} = \mathbb {E} _ {t, \boldsymbol {\epsilon}, c} \left[ \sigma_ {t} \omega (t) \left(\nabla_ {\boldsymbol {x} _ {t}} \log p _ {t} \left(\boldsymbol {x} _ {t} \mid y ^ {c}\right) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} ^ {\mu_ {\tau}} \left(\boldsymbol {x} _ {t} \mid c, y\right)\right) \frac {\partial \boldsymbol {g} (\theta , c)}{\partial \theta} \right]. \tag {33}
$$
D Additional Ablation Study
D.1 Ablation Study on Large Scene Generation
Here we perform an ablation study on large scene generation to validate the effectiveness of our proposed improvements. We start from 64 rendering resolution, with SDS loss and our scene initialization. The results are shown in Figure 11. It can be seen from the figure that, with our scene initialization, the results are with  $360^{\circ}$  surroundings instead of being object-centric. Increasing rendering resolution is slightly beneficial. Adding annealed time schedule improves the visual quality of the results. Replacing SDS with VSD makes the results more realistic with more details.
Figure 11: Ablation study on scene generation. With our scene initialization, the results are with  $360^{\circ}$  surroundings instead of being object-centric. Our annealed time schedule and VSD are both beneficial for the generation quality. The prompt here is Inside of a smart home, realistic detailed photo, 4k.
D.2 Ablation Study on Mesh Fine-tuning
Here we provide an ablation study on mesh fine-tuning. Fine-tuning with textured mesh further improves the quality compared to the NeRF result. Fine-tuning texture with VSD provides higher fidelity than SDS. Note that both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results by our VSD. And it's crucial to get a high-quality NeRF with VSD at the first stage.
D.3 Ablation on Number of Particles
Here we provide ablation study on number of particles. We vary the number of particles in 1, 2, 4, 8 and examine how the number of particles affects the generated results. The CFG of VSD is set as 7.5. The results are shown in Fig. 13. As is shown in the figure, the diversity of the generated results is slightly larger as the number of particles increases. Meanwhile, the quality of generated results is not affected much by the number of particles. Owing to the high computation overhead to optimize 3D representations and limitations on computation resources, we now only test at most 8 particles. We provide a 2D experiment with 2048 particles in Appendix G to demonstrate the scalability of VSD. We leave the experiments of more particles in 3D as future work.
(1) High fidelity NeRF generated by VSD (ours).
(2) Extract and fine-tune geometry from NeRF.
Figure 12: Pipeline of ProlificDreamer along with ablation study of VSD. After generating a high-quality NeRF, we extract and finetune a textured mesh. VSD provides high-fidelity texture, while SDS tends to generate smoother results. Note that both VSD and SDS in mesh fine-tuning are based on and benefit from the high-fidelity NeRF results by our VSD. The prompt here is an elephant skull.
(3a) Texture fine-tuned by VSD (ours).
(3b) Texture fine-tuned by SDS.
Algorithm 1 Variational Score Distillation
Input: Number of particles  $n$  ( $\geq 1$ ). Large text-to-image diffusion model  $\epsilon_{\mathrm{pretrain}}$ . Learning rate  $\eta_1$  and  $\eta_2$  for 3D structures and diffusion model parameters, respectively. A prompt  $y$ .
1: initialize  $n$  3D structures  $\{\theta^{(i)}\}_{i=1}^n$ , a noise prediction model  $\epsilon_{\phi}$  parameterized by  $\phi$ .
2: while not converged do
3: Randomly sample  $\theta \sim \{\theta^{(i)}\}_{i = 1}^n$  and a camera pose  $c$ .
4: Render the 3D structure  $\theta$  at pose  $c$  to get a 2D image  $\pmb{x}_0 = \pmb{g}(\theta, c)$ .
5:  $\theta \gets \theta -\eta_1\mathbb{E}_{t,\pmb {\epsilon},c}\left[\omega (t)\left(\pmb{\epsilon}_{\mathrm{pretrain}}(\pmb {x}_t,t,y^c) - \pmb{\epsilon}_{\phi}(\pmb {x}_t,t,c,y)\right)\frac{\partial\pmb{g}(\theta,c)}{\partial\theta}\right]$
6:  $\phi \gets \phi -\eta_{2}\nabla_{\phi}\mathbb{E}_{t,\epsilon}||\pmb{\epsilon}_{\phi}(\pmb{x}_t,t,c,y) - \pmb {\epsilon}||_2^2.$
7: end while
8: return
D.4 Ablation on Rendering Resolution
Here we provide an ablation study on the rendering resolution during NeRF training with VSD. As shown in Fig. 14, training with a higher resolution produces better results with finer details. In addition, our VSD still provides competitive results under a lower training resolution (128 or 256), which is more computationally efficient than the 512 resolution.
E Algorithm for Variational Score Distillation
We provide a summarized algorithm of variational score distillation in Algorithm 1.