SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While diffusion models prove effective for generative modeling, it often takes a large number of discretization steps in the backward process (9) to produce high quality samples, which caps their potential for real time applications. Note that the forward process (8) naturally provides another type of auxiliary bridge, which combined with HSIVI, can be used to accelerate the sampling process of diffusion models.
Example 2 (Diffusion Bridge). Consider the forward process  $\{\pmb{u}_s\}_{s\in [0,L]}$  with  $L > 0$  (defined in equation (8)) in diffusion models. We choose  $T$  discrete time steps  $0\approx s_0 < \dots < s_{T - 1}\leq L$  and
let  $\pmb{x}_t\coloneqq \pmb{u}_{s_t}$  with probability density function  $p_t(\cdot)$ . Assume each auxiliary distributions  $p_t(\cdot)$  for  $0\leq t\leq T - 1$  admits a score function as
$$
\boldsymbol {S} _ {t} (\boldsymbol {x}) := \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) \approx \boldsymbol {S} ^ {*} (\boldsymbol {x}, s _ {t}), 0 \leq t \leq T - 1,
$$
where  $S^{*}(\pmb{x}, s)$  is a pre-trained score model with the denoising score matching loss (equation (13) in Appendix A). Let us denote  $S^{*}(\pmb{x}, s_{t})$  by  $S_{t}^{*}(\pmb{x})$  for short. With sufficient samples from the data distribution  $p_{0}(\pmb{x})$  and model capacity, the approximation  $S_{t}^{*}(\pmb{x})$  can be reasonably accurate for almost all  $\pmb{x}$  and  $t$  (Song et al., 2020b).
As the pre-trained score model provides a diffusion bridge from the simple distribution  $p_{T-1}$  (e.g., standard Gaussian) to the data distribution, we can train the hierarchical semi-implicit distributions to approximate the diffusion bridge within the HSIVI framework. Given the expressiveness of hierarchical semi-implicit distributions, we may expect an accurate approximation of the data distribution with a small number  $T$  and hence acceleration can be achieved.
However, the memory usage during the sequential training process for HSIVI might be large because of the necessity for independent parameters. Therefore, we may employ a parameter sharing scheme which is commonly assumed in diffusion models (Song & Ermon, 2019; Ho et al., 2020) such that different conditional layers share the same parameters  $\phi$ . Note that sequential training is not suitable in this setting. Therefore, we propose a joint training procedure that minimizes a weighted sum of the SIVI objectives
$$
\mathcal {L} _ {\mathrm {H S I V I -} f} (\phi) = \sum_ {t = 0} ^ {T - 1} \beta (t) \mathcal {L} _ {\mathrm {S I V I -} f} \left(p _ {t} \left(\boldsymbol {x} _ {t}\right) \| q _ {t} \left(\boldsymbol {x} _ {t}; \phi\right)\right), \tag {10}
$$
where  $\beta(t): \{0, \dots, T-1\} \to \mathbb{R}_+$  is a positive weighting function and  $f$  is some distance criterion. See Algorithm 2 in Appendix C.3 for more details of joint training.
More specifically, in this work, we mainly focus on building the diffusion bridge with variance preserving SDE (VP-SDE) (Song et al., 2020b) such that  $\pmb{u}_s|\pmb{u}_0 \sim \mathcal{N}(\sqrt{\alpha(s)}\pmb{u}_0, (1 - \alpha(s))\pmb{I})$  with a decreasing function  $\alpha(s)$  of  $s$ . We use  $\mathcal{L}_{\mathrm{HSIVI-SM}}$  in equation (10) for training and set the weighting function  $\beta(t) = 1 - \alpha(s_t)$  as recommended in Song et al. (2020b), which tends to train layers that are far from  $t = 0$  first during the training, resembling the sequential training. Another popular formulation of diffusion models is to fit a noise model  $\epsilon^*(\pmb{x}, s)$  that predicts the noise added to a noisy sample  $\pmb{x}$  at time  $s$  (Ho et al., 2020). HSIVI-SM also generalizes to the case where a pre-trained noise model is available. The pre-trained noise model forms a (generalized) diffusion bridge by letting  $\pmb{\epsilon}_t^*(\pmb{x}) = \pmb{\epsilon}^*(\pmb{x}, s_t)$ , and we call the corresponding training method "ε-training". We provide a reparametrized objective function  $\tilde{\mathcal{L}}_{\mathrm{HSIVI-SM}}$  for  $\epsilon$ -training in Appendix C.4.
Several efforts have been made to accelerate the sampling process of diffusion models, including faster numerical ordinary differential equation (ODE) solvers (Song et al., 2020a; Zhang & Chen, 2022; Lu et al., 2022) and distillation techniques (Luhman & Luhman, 2021; Salimans & Ho, 2022; Zheng et al., 2022). Our approach is different from these previous efforts in that we accelerate the stochastic diffusion model directly (hence would provide more diverse samples (Figure 6)) and do not require sampling datasets from the diffusion models prior to distillation which is computationally expensive. From a Bayesian perspective, HSIVI is related to Song & Ermon (2019), where the authors used the annealed Langevin dynamics guided by a pre-trained score model to sample from the data distribution. By solving this problem using a variational inference approach, HSIVI enjoys faster sampling speed and scales better to high-dimensional data.
In this section, we first compare HSIVI to its single-layer counterpart, SIVI, on two inference tasks. We use the sequential training method where each conditional layer in the hierarchical semi-implicit variational distributions has independent parameters. We then apply HSIVI-SM to diffusion model acceleration on various datasets. As the memory consumption for generative models is large, we use the joint training method where the conditional layers in hierarchical semi-implicit distributions have shared parameters across different  $ts$ . For all experiments, each conditional layer is modeled as a Gaussian distribution with parametrized mean and variance. More details of the model architectures and hyper-parameters are included in Appendix E. The code is available at https://github.com/longinYu/HSIVI.
Figure 2: Comparison of 10,000 generated samples from SIVI and 5-layer HSIVI on a two-dimensional Gaussian mixture model (blue).
Figure 3: The posterior estimates obtained by different methods. For each method, we collect 100,000 samples to calculate the sample mean and confidence interval.
5.1 Target distribution approximation
Gaussian mixture model We first evaluate HSIVI and SIVI on a two-dimensional Gaussian mixture model. The target distribution  $p(\pmb{x})$  takes the form  $p(\pmb{x}) = \sum_{i=1}^{8} 1/8 \cdot \mathcal{N}(\pmb{x}; \pmb{\mu}_i, \sigma^2\mathbf{I})$  where  $\pmb{\mu}_i = [10\cos(\frac{i\pi}{4}), 10\sin(\frac{i\pi}{4})]^T$ ,  $\sigma = 1$ . For HSIVI, we construct an auxiliary bridge of  $T = 5$  with geometric interpolation in Example 1, where  $p_{\mathrm{base}}(\pmb{x}) = \mathcal{N}(\pmb{x}; \mathbf{0}, \mathbf{I})$  and  $\lambda_t = 1 - t/5$ . The results are presented in Figure 2. Note that the modes in this Gaussian mixture model are far apart from each other, and both SIVI-LB and SIVI-SM are trapped in local modes. In contrast, both HSIVI-LB and HSIVI-SM discover all modes and provide an accurate approximation of the target distribution with HSIVI-SM being better for recovering the right scale of variance.
High-dimensional conditioned diffusion The second example is a high-dimensional Bayesian inference problem arising from the following Langevin SDE
$$
\mathrm {d} x _ {s} = 1 0 x _ {s} \left(1 - x _ {s} ^ {2}\right) \mathrm {d} s + \mathrm {d} w _ {s}, \tag {11}
$$
where  $x_0 = 0$  and  $w_{s}$  is a one-dimensional standard Brownian motion. This system describes the motion of a particle with negligible mass trapped in an energy potential with thermal fluctuations represented by the Brownian forcing (Cui et al., 2016). Using an Euler-Maruyama scheme with step size  $\Delta s = 0.01$  on a time interval [0,3], we discretize the SDE (11) into  $\pmb {x} = (x_{d_1},\dots ,x_{d_{300}})$  where  $d_{i} = 0.01i$ , which gives the prior distribution  $p_{\mathrm{prior}}(\pmb {x})$  of the 300-dimensional variable  $\pmb{x}$ . The noisy observations  $\pmb{y}$  is obtained by  $\pmb {y} = \pmb {x} + \pmb{\xi}$ , where  $\pmb {\xi}\sim \mathcal{N}(\mathbf{0},\sigma^2\mathbf{I})$  with  $\sigma = 0.1$ . Our goal is to infer the posterior distribution of the latent states  $p(\pmb {x}|\pmb {y})\propto p_{\mathrm{prior}}(\pmb {x})p(\pmb {y}|\pmb {x})$ . The ground truth is formed by running 100,000 independent stochastic gradient Langevin dynamics (SGLD) chains with a step size of 0.0001 and collecting the results after 10,000 iterations.
For HSIVI, we form the auxiliary bridge using geometric interpolation with  $p_{\mathrm{base}}(\pmb{x}) = \mathcal{N}(\pmb{x};\pmb{y},\sigma^2\mathbf{I})$  and  $\lambda_t = 1 - t / (T - 1)$  for  $t = 0,\dots ,T - 1$ . Figure 3 shows the estimated posteriors obtained by different methods. We see that SIVI-SM severely underestimates the variance. With  $T = 5$  layers, HSIVI-SM fits the variance better and hence provides more accurate posterior estimates. For both HSIVI-SM and HSIVI-LB, the estimated covariance matrix becomes more accurate as  $T$  increases (Table 4 in Appendix D.2), demonstrating the effectiveness of hierarchical models for fitting complicated distributions.
Figure 4: Sample trajectories generated from 10-layer HSIVI-SM on four 2D toy examples. The arrows represent the estimated score function in HSIVI-SM. The sample size is 10,000.
Table 1: JS divergences between the target distribution and the variational approximation on the four toy datasets. The results of HSIVI-SM are averaged by 5 independent runs with standard deviation in the subscripts. JS divergences are calculated by the ITE package (Szabó, 2014) with 10,000 samples.
Name	T=5	T=10	T=1000
DDPM	DDIM	HSIVI-SM	DDPM	DDIM	HSIVI-SM	DDPM
Checkerboard	0.891	0.591	0.068±0.006	0.521	0.373	0.030±0.005	0.058
Swissroll	1.037	0.332	0.126±0.006	0.334	0.164	0.082±0.003	0.042
Circles	0.907	0.397	0.083±0.015	0.364	0.201	0.073±0.005	0.032
Moons	0.961	0.355	0.096±0.013	0.352	0.137	0.059±0.007	0.036
5.2 Diffusion model acceleration
2D toy examples In this toy model example, we test four synthetic 2D datasets: Checkerboard, Circles, Moons, and Swissroll (Pedregosa et al., 2011). We first pre-train the score model  $S^{*}(\boldsymbol{x}, s)$  for  $s \in [0,1]$  with quadratic noise schedule  $1 - \alpha(s) = s^2$ . For constructing the  $T$ -layer diffusion bridge, we select  $\{s_t\}_{t=0}^{T-1}$  so that  $1 - \alpha(s_t) = [0.01 + (\sqrt{0.8} - 0.01)t/T]^2$ . Figure 4 shows the sample trajectories  $(\boldsymbol{x}_9, \boldsymbol{x}_7, \boldsymbol{x}_5$  and  $\boldsymbol{x}_0)$  progressively generated from 10-layer HSIVI-SM. We see clearly how the semi-implicit distributions are guided towards the target distribution and all modes are discovered. We also report the Jensen-Shannon (JS) divergence between the target distributions and the estimated distributions in Table 1. We see that HSIVI-SM significantly improves upon DDIM and DDPM in both cases with 5 and 10 steps. Also, 10-layer HSIVI-SM is comparable to DDPM with 1000 full steps. See Figure 10 in Appendix D.3 for visualization of samples from different methods.
MNIST On MNIST, we use the noise model  $\epsilon^{*}(\pmb{x}, s)$  instead of the score model and use  $\epsilon$ -training to train HSIVI-SM. The structure of  $\epsilon^{*}(\pmb{x}, s)$  follows the UNet in Ho et al. (2020) by reducing the number of input and output channels to one. With the same noise schedule employed in Song et al. (2020a), we first pre-train the noise model  $\epsilon^{*}(\pmb{x}, s)$  with 1000 discretization steps and then form the  $T$ -layer diffusion bridge for HSIVI-SM by selecting  $T$  discrete time steps. Figure 5 shows the samples from DDPM, DDIM, and HSIVI-SM with  $T = 5$  steps. We see that the samples produced by HSIVI-SM are much cleaner and more recognizable than those produced by DDPM and DDIM.
CIFAR-10, CelebA & ImageNet On both CIFAR-10 and CelebA, the structure of our pre-trained noise model  $\epsilon^{*}(\boldsymbol{x}, s)$  follows the UNet structure(Ronneberger et al., 2015) employed by Ho et al. (2020), instead of the huge VP deep continuous-time model (Song et al., 2020b) that has more channels and layers. We also provide additional results on ImageNet  $(64 \times 64)$  with more powerful pre-trained score nets in (Nichol & Dhariwal, 2021)(biger models with more parameters). Since this generative modeling has been formulated as a score-based VI problem, we do not have to use any training data for training HSIVI-SM.
Following the noise schedule employed in Song et al. (2020a), we first pre-train the noise model  $\epsilon^{*}(\pmb{x},s)$  with 1000 discretization steps and then form the  $T$ -layer diffusion bridge for HSIVI-SM
DDPM  $(T = 5)$
DDIM  $(T = 5)$
Figure 5: Comparison of the quality of uncurated samples generated by DDPM, DDIM, and HSIVISM with 5 discrete time steps on MNIST.
HSIVI-SM  $(T = 5)$
Table 2: Sample quality measured by FID (↓) on CIFAR-10, CelebA and ImageNet with a varying number of function evaluations (NFE). Results of baselines are calculated by running their official codes, where the architectures of score model (or noise model) are the UNet employed in Ho et al. (2020) for CIFAR-10 and CelebA and (Nichol & Dhariwal, 2021) in ImageNet.
Dataset	CIFAR-10 (32×32)	CelebA (64×64)	ImageNet (64×64)
NFE	5	10	15	5	10	15	5	10	15
DDPM (Ho et al., 2020)	320.16	278.65	198.00	366.10	309.95	206.92	402.68	358.80	284.00
DDIM (Song et al., 2020a)	41.53	13.73	8.78	27.38	10.89	7.78	147.03	42.31	24.85
FastDPM (Kong & Ping, 2021)	67.64	9.85	6.16	27.63	15.44	12.05	N/A	N/A	N/A
Analytic-DDPM (Bao et al., 2022)	93.16	34.54	20.03	50.92	28.93	21.84	N/A	60.65	45.98
Analytic-DDIM (Bao et al., 2022)	51.86	14.08	8.65	29.40	15.74	12.25	N/A	70.62	41.56
DPM-Solver-fast (Lu et al., 2022)	329.13	10.89	4.67	355.96	6.76	2.98	402.43	28.96	20.03
HSIVI-SM (ours)	6.27	4.31	4.17	6.22	3.09	2.23	40.43	17.67	15.49
by selecting  $T$  discrete time steps as before. For HSIVI-SM with  $\epsilon$ -training, the conditional layer  $q_{t}(\cdot|\boldsymbol{x}_{t+1};\phi)$  is modeled as a Gaussian distribution with mean  $\mu_{t}(\boldsymbol{x}_{t+1};\phi^{\mu})$  and diagonal variance matrix  $\Sigma_{t}(\phi^{\sigma})$  where  $\{\phi^{\mu},\phi^{\sigma}\} = \phi$  are the variational parameters. In our implementations, both  $\mu_{t}(\boldsymbol{x}_{t+1};\phi^{\mu})$  and  $\boldsymbol{f}_{t}(\boldsymbol{x}_{t};\psi)$  use the same architecture as  $\epsilon^{*}(\boldsymbol{x},s)$ . The number of layers, which is also the number of function evaluations (NFE), is set to be  $T = 5,10,15$  in our experiments. We train HSIVI-SM with the same setting for  $T = 10,15$ . The 5-layer HSIVI-SM is trained by further fine-tuning the well-trained 15-layer HSIVI-SM and we find this strategy leads to better results. During each nested training loop of  $\boldsymbol{f}_{t}(\boldsymbol{x}_{t};\psi)$ , we update  $\psi$  20 times before each update of  $\phi$ , since we find  $\boldsymbol{f}_{t}(\boldsymbol{x}_{t};\psi)$  needs more training empirically to provide reliable guidance.
For each method, we draw 50,000 samples and use the Fréchet inception distance (FID) score (Karras et al., 2022) to evaluate the sample quality (Table 2). We find that HSIVI-SM performs on par or better than the other baselines on both CIFAR-10 and CelebA, and the advantage is evident when the NFE is small. The sampling trajectories of 10-layer HSIVI-SM on CelebA with the same starting point but different random seeds are shown in Figure 6. We see that HSIVI-SM is capable of producing more diverse samples due to its stochastic nature, which is different from existing ODE based fast diffusion model samplers.
Figure 6: Sample trajectories of 10-layer HSIVISM with the same starting point  $\pmb{x}_{10}$  on CelebA.
5.3 Additional Study
Ablation of layers number In Figure 7, We provide a failure case on fitting the checkerboard target with diffusion bridge, demonstrating that the HSIVI-SM algorithm fails when the layer number  $T$  is small (the distances of auxiliary distributions at successive time steps are large) on a checkerboard
Figure 7: Failure cases of HSIVI-SM. The quivers show the estimated score by the  $f$  function.  $T$  is the layers number of HSIVI-SM. The generated samples in orange show that smaller  $T$  may fail on this example.
Table 3: Comparison of non-isotropic conditional layers and isotropic conditional layers on CIFAR10, the sample quality is measured by FID (↓).
NFE	DDPM	DDIM	HSIVI-SM (isotropic)	HSIVI-SM (non-isotropic)
5	320.16	41.53	7.33	6.27
10	278.65	13.73	4.78	4.31
15	198.00	8.78	4.46	4.17
distribution. In fact, the score function on the checkerboard target is sharp on the boundaries but vanishes elsewhere. Therefore, fitting this target distribution is somewhat challenging.
Ablation of the variational family To validate the improvement of HSIVI-SM on diffusion models, we train HSIVI-SM with isotropic conditional layers in consistency with denoising-diffusion sampling, like DDPM and DDIM. We report the results of FID on the CIFAR-10 dataset in Table 3. These results provide further evidence for the statement outlined in Section 3.2. HSIVI-SM matches the marginal distributions  $q_{t}(\boldsymbol{x}_{t})$  and  $p_{t}(\boldsymbol{x}_{t})$  directly via score matching and would ensure a better fit for  $p_{0}(\boldsymbol{x}_{0})$ . The enhancement of HSIVI-SM over DDPM stems not only from its more expressive variational distribution but also from the direct alignment of the marginal distributions.
We introduced HSIVI, a hierarchical semi-implicit variational inference method that enables more expressive multi-layer construction of semi-implicit distributions. Given appropriate auxiliary distributions that interpolate between a simple base distribution and the target distribution, the conditional layers in hierarchical semi-implicit distributions can be progressively trained one layer after another. In experiments, we showed that HSIVI outperforms previous single-layer SIVI methods on several Bayesian inference tasks with complicated posteriors. HSIVI can also be used to accelerate the sampling process of diffusion models, where pre-trained score networks serve as a natural sequence of bridging distributions, which allows for direct acceleration of the stochastic diffusion model and does not require expensive sampling from the diffusion models during training. We showed that HSIVI can produce high quality samples comparable to or better than existing fast diffusion model samplers with few function evaluations on various datasets. Limitations are discussed in Appendix F.
This work was supported by National Natural Science Foundation of China (grant no. 12201014 and grant no. 12292983). The research of Cheng Zhang was supported in part by National Engineering Laboratory for Big Data Analysis and Applications, the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. The authors appreciate the anonymous NeurIPS reviewers for their constructive feedback.
L. K. Saul and M. I. Jordan. Exploiting tractable substructures in intractable networks. In Advances in Neural Information Processing Systems, 1996.
J. Shi, S. Sun, and J. Zhu. Kernel implicit variational inference. In International Conference on Learning Representations, 2018a.
J. Shi, S. Sun, and J. Zhu. A spectral approach to gradient estimation for implicit distributions. In International Conference on Machine Learning, 2018b.
Artem Sobolev and Dmitry P. Vetrov. Importance weighted hierarchical variational inference. In Neural Information Processing Systems, 2019.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Y. Song, S. Garg, J. Shi, and S. Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the Thirty-Fifth Conference on Uncertainty in Artificial Intelligence, 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Zoltán Szabó. Information theoretical estimators toolbox. Journal of Machine Learning Research, 15:283-287, 2014.
Michalis K. Titsias and Francisco J. R. Ruiz. Unbiased implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 167-176. PMLR, 2019.
D. Tran, D. M. Blei, and E. M. Airoldi. Copula variational inference. In Advances in Neural Information Processing Systems, 2015.
D. Tran, R. Ranganath, and D. M. Blei. Hierarchical implicit models and likelihood-free variational inference. In Advances in Neural Information Processing Systems, 2017.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661-1674, 2011. doi: 10.1162/NECO_a_00142.
M. J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Maching Learning, 1(1-2):1-305, 2008.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5646-5655, 2018.
Longlin Yu and Cheng Zhang. Semi-implicit variational inference via score matching. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=sd90a2yptrt.
C. Zhang, B. Shahbaba, and H. Zhao. Variational hamiltonian monte carlo via score matching. Bayesian Analysis, 13(2):485-506, 2018.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
A Details of diffusion models
Diffusion models work by adding noise to the training data in the forward process and then removing the noise to recover the data in the backward process, which can be integrated into a general stochastic differential equation (SDE) framework (Song et al., 2020b). The forward process  $\{\pmb{u}_s\}_{s\in [0,L]}$  is usually described by the SDE
$$
\mathrm {d} \boldsymbol {u} _ {s} = \boldsymbol {f} (\boldsymbol {u} _ {s}, s) \mathrm {d} s + g (s) \mathrm {d} \boldsymbol {w} _ {s}, \quad \boldsymbol {u} _ {0} \sim p _ {0} (\cdot),
$$
where  $p_0(\cdot)$  is the data distribution,  $w_{s}$  is a standard Brownian motion,  $f(u_{s}, s)$  and  $g(s)$  are the drift and diffusion coefficient respectively. To generate samples from the data distribution, one can run the following reversed SDE
$$
\mathrm {d} \boldsymbol {u} _ {s} = \left[ \boldsymbol {f} \left(\boldsymbol {u} _ {s}, s\right) - g ^ {2} (s) \nabla_ {\boldsymbol {u} _ {s}} \log p _ {s} \left(\boldsymbol {u} _ {s}\right) \right] \mathrm {d} s + g (s) \mathrm {d} \bar {\boldsymbol {w}} _ {s}, \quad \boldsymbol {u} _ {L} \sim p _ {L} (\cdot),
$$
where  $p_{s}(\cdot)$  is the probability density function (pdf) of  $\mathbf{u}_s$  and  $\bar{\boldsymbol{w}}_s$  is a standard Brownian motion when time flows from  $L$  to 0. There exists deterministic process shares the same marginal probability densities  $\{p_s(\cdot)\}_{s\in [0,L]}$  described by the following ordinary differential equation (ODE)
$$
\mathrm {d} \boldsymbol {u} _ {s} = [ \boldsymbol {f} (\boldsymbol {u} _ {s}, s) - \frac {1}{2} g ^ {2} (s) \nabla_ {\boldsymbol {u} _ {s}} \log p _ {s} (\boldsymbol {u} _ {s}) ] \mathrm {d} s, \quad \boldsymbol {u} _ {L} \sim p _ {L} (\cdot),
$$
called probability flow (PF) ODE.