Diffusion Models as Cartoonists! The Curious Case of High Density Regions

## Abstract

Abstract We investigate what kind of images lie in the high-density regions of diffusion models. We introduce a theoretical mode-tracking process capable of pinpointing the exact mode of the denoising distribution, and we propose a practical high-probability sampler that consistently generates images of higher likelihood than usual samplers. Our empirical findings reveal the existence of significantly higher likelihood samples that typical samplers do not produce, often manifesting as cartoon-like drawings or blurry images depending on the noise level. Curiously, these patterns emerge in datasets devoid of such examples. We also present a novel approach to track sample likelihoods in diffusion SDEs, which remarkably incurs no additional computational cost.

## 1 Introduction

Recently, Karras et al. () attributed the empirical success of guided diffusion models to their ability to limit outliers, i.e. samples ${\bm{x}}_{0}\sim p_{0}$ with low likelihood $p_{0}({\bm{x}}_{0})$.
We argue that these models in fact also elude samples with very high likelihoods. Our assertion stems from investigating a key question: what manifests if we bias the sampler towards high-density regions of $p_{0}$?
However, an immediate hurdle is the inability of stochastic diffusion models to track their own likelihood (Song et al., ; ).

Figure: Figure 1: High-density samples ${\bm{y}}_{0}|{\bm{x}}_{t}$ resemble cartoon drawings. This is in contrast to regular denoising samples ${\bm{x}}_{0}\sim p({\bm{x}}_{0}|{\bm{x}}_{t})$ or expectations $\mathbb{E}[{\bm{x}}_{0}|{\bm{x}}_{t}]$. The data contains no cartoons.
Refer to caption: extracted/5973435/figures/ffhq-high-prob-vs-mean-changed-label.png

First, we show that likelihood can be tracked in diffusion models with novel augmented stochastic differential equations (SDE), which govern the evolution of a sample with its log-density $\log p_{t}({\bm{x}}_{t})$ under the optimal (unknown) model. For approximate models, we provide a formula for the bias in the log-density estimate. The evaluation of the log-density estimate comes at no additional cost, and can be used with any pretrained model without further tuning.

Then, we introduce a theoretical mode-tracking process, which finds the exact mode of the denoising distribution $p({\bm{x}}_{0}|{\bm{x}}_{t})$ under some technical assumptions, albeit at a high computational cost. We propose an approximative *high probability sampler* whose images almost always have higher likelihood than regular samples.

We leverage these findings to give insights into the diffusion model probability landscape. We find that dramatically higher likelihood samples exist that a regular sampler never returns in practise. We observe that the high density samples tend to be (i) blank images for high noise levels, (ii) cartoon drawings for moderate noise, and (iii) blurry images for low noise (See Figure 1).
*This is despite the datasets not containing any cartoon drawings.*
Curiously, on FFHQ-256 we observe 97% correlation between model’s likelihood estimates and the amount of information in the image.

We summarize our contributions in Figure 2.

Figure: Figure 2: Our contributions.

## 2 Likelihood estimation in SDE models

We model the data distribution with a diffusion process (Song et al., ), which gradually transforms the data into a Gaussian. We define a forward process $p({\bm{x}}_{t}|{\bm{x}}_{0})=\mathcal{N}({\bm{x}}_{t};\alpha_{t}{\bm{x}}_{0},
\sigma_{t}^{2}{\bm{I}}_{D})$, where $D$ is the dimensionality of the data, $\alpha_{t}$ and $\sigma_{t}$ are schedule parameters with a decreasing signal-to-noise ratio
$\text{SNR}(t)=\alpha_{t}^{2}/\sigma_{t}^{2}=e^{\lambda_{t}}$.
Song et al. () showed an equivalence to a linear stochastic differential equation (SDE) with ${\bm{x}}_{0}\sim p_{0}$ - data distribution and

$$ $d{\bm{x}}_{t}=f(t){\bm{x}}_{t}dt+g(t)d{\textnormal{W}}_{t},$ (1) $$

with drift $f(t)=\frac{d\log\alpha_{t}}{dt}$, diffusion $g^{2}(t)=-\frac{d\lambda_{t}}{dt}\sigma_{t}^{2}$, and $\{{\textnormal{W}}_{t}\}$ a Wiener process.
Most of our theoretical results also hold for non-linear SDEs such as the one proposed by Bartosh et al. (). We discuss the impact of the linearity of the drift in section 6 and provide the general formulas in the appendices.
Anderson () showed that there exists a corresponding (sharing the distribution over trajectories) reverse-time SDE

$$ $d{\bm{x}}_{t}=\Big{(}f(t){\bm{x}}_{t}-g^{2}(t)\definecolor{tcbcolback}{rgb}{ 1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{ rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to53.73pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt \lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.45787pt}{16 .84525pt}\pgfsys@curveto{51.81499pt}{16.84525pt}{53.72575pt}{14.93448pt}{53.72 575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{4.26788pt}\pgfsys@curveto{53.7257 5pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15 .42262pt}{52.30313pt}{14.14877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.303 13pt}{4.26788pt}\pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{ 49.45787pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\Big{)}dt+g(t)d \overline{{\textnormal{W}}}_{t},$ (2) $$

where $p_{t}$ is the marginal density of the forward process (equation 1), and $\{\overline{{\textnormal{W}}}_{t}\}$ is a Wiener process going backwards in time.
Training of diffusion models centers on efficient estimation of the *score* $\nabla_{\bm{x}}\log p_{t}({\bm{x}})$ (Hyvärinen & Dayan, ; Vincent, ; Song et al., ; Kingma & Gao, ).

### 2.1 Diffusion ODE

We are interested in sampling high-probability regions of diffusion models and for that, we need ability to calculate the likelihood $p_{0}({\bm{x}})$ of a sample image ${\bm{x}}\sim p_{0}$.
As the reverse-SDE (equation 2) does not track the likelihood evolution $d\log p_{t}({\bm{x}}_{t})$, a typical method for obtaining sample likelihoods is instead via the Probability-Flow ODE (Chen et al., ; Song et al., )

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \log p_{t}({\bm{x}}_{t})\end{bmatrix}=\begin{bmatrix}f(t){\bm{x}}_{t}-\frac{1} {2}g^{2}(t)\definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe} {rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\ -f(t)D+\frac{1}{2}g^{2}(t)\definecolor{tcbcolback}{rgb}{1,0.85,0.85} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to23.66 pt{\vbox to15pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{10.72736pt}\pgfsys@curveto{0.0pt}{13.08447pt}{1.91077pt} {14.99524pt}{4.26788pt}{14.99524pt}\pgfsys@lineto{19.38907pt}{14.99524pt} \pgfsys@curveto{21.74619pt}{14.99524pt}{23.65695pt}{13.08447pt}{23.65695pt}{10 .72736pt}\pgfsys@lineto{23.65695pt}{4.26788pt}\pgfsys@curveto{23.65695pt}{1.91 077pt}{21.74619pt}{0.0pt}{19.38907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{10.7273 6pt}\pgfsys@curveto{1.42262pt}{12.29877pt}{2.69647pt}{13.57262pt}{4.26788pt}{1 3.57262pt}\pgfsys@lineto{19.38907pt}{13.57262pt}\pgfsys@curveto{20.96048pt}{13 .57262pt}{22.23433pt}{12.29877pt}{22.23433pt}{10.72736pt}\pgfsys@lineto{22.234 33pt}{4.26788pt}\pgfsys@curveto{22.23433pt}{2.69647pt}{20.96048pt}{1.42262pt}{ 19.38907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{4.62817pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\operatorname{div}_{\bm{x}}$}}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{} \hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss }}\lxSVG@closescope\endpgfpicture}}\definecolor{tcbcolback}{rgb}{1,0.91,0.82} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.45787pt}{16.84525pt} \pgfsys@curveto{51.81499pt}{16.84525pt}{53.72575pt}{14.93448pt}{53.72575pt}{12 .57736pt}\pgfsys@lineto{53.72575pt}{4.26788pt}\pgfsys@curveto{53.72575pt}{1.91 077pt}{51.81499pt}{0.0pt}{49.45787pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15 .42262pt}{52.30313pt}{14.14877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.303 13pt}{4.26788pt}\pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{ 49.45787pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\end{bmatrix}dt,$ (3) $$

where $\operatorname{div}_{\bm{x}}=\sum_{i}\frac{\partial}{\partial x_{i}}$ is the divergence operator. The PF-ODE is a continuous normalizing flow (CNF) that shares marginal distributions $p_{t}({\bm{x}}_{t})$ with both forward (1) and reverse SDE (2), when they share $p_{0}$ (Song et al., ). The PF-ODE tracks and returns both the sample ${\bm{x}}_{0}$ and its likelihood $\log p_{0}({\bm{x}}_{0})$ from ${\bm{x}}_{T}\sim p_{T}$. It requires second-order derivatives $\operatorname{div}\nabla$, which are roughly twice as expensive as the score to evaluate (Hutchinson, ; Grathwohl et al., ).

### 2.2 Augmented stochastic dynamics

Figure: Figure 3: Tracking stochastic sampling likelihood. Estimation of $\log p_{t}({\bm{x}}_{t})$ (colored trajectory) for stochastic sampling via Augmented Reverse SDE (equation 4) on a Gaussian mixture with known $\nabla_{\bm{x}}\log p_{t}({\bm{x}})$ and $p_{T}$. Evaluation of $d\log p_{t}({\bm{x}}_{t})$ requires only the score function.
Refer to caption: extracted/5973435/figures/updated_fig1_logp_fixed_eqrefs.png

It has been reported that a stochastic sampler (2) has superior sample quality to the PF-ODE (3) (Song et al., ; Karras et al., ). However, the absence of density-augmented SDEs forbids the stochastic sampler from describing the likelihood of the samples it yields (Song et al., ; Lu et al., ; Zheng et al., ; Lai et al., ).
We bridge this gap by generalizing equation 3 to account for the stochastic evolution of ${\bm{x}}$.

###### Theorem 1 (Augmented reverse SDE) .

Let ${\bm{x}}$ be a random process defined by equation 2. Then

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \log p_{t}({\bm{x}}_{t})\end{bmatrix}=\begin{bmatrix}f(t){\bm{x}}_{t}-g^{2}(t) \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\ -f(t)D-\frac{1}{2}g^{2}(t)\|\definecolor{tcbcolback}{rgb}{1,0.91,0.82} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.45787pt}{16.84525pt} \pgfsys@curveto{51.81499pt}{16.84525pt}{53.72575pt}{14.93448pt}{53.72575pt}{12 .57736pt}\pgfsys@lineto{53.72575pt}{4.26788pt}\pgfsys@curveto{53.72575pt}{1.91 077pt}{51.81499pt}{0.0pt}{49.45787pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15 .42262pt}{52.30313pt}{14.14877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.303 13pt}{4.26788pt}\pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{ 49.45787pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\|^{2}\end{bmatrix}dt +g(t)\begin{bmatrix}{\bm{I}}_{D}\\ \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d \overline{{\textnormal{W}}}_{t}.$ (4) $$

The proof (Appendix E) consists of applying Itô’s lemma and the Fokker-Planck equation.
Note that $d\log p_{t}({\bm{x}}_{t})$ is economical to track as it only needs access to the first-order score. Figure 3 visualises a particle following a stochastic reverse trajectory while tracking its log-density.
This result contributes a new useful and economical tool to generate a sample together with its density estimate, which previously was done using PF-ODE (Jing et al., ).
Similarly to the reverse SDE, we also introduce density-tracking forward SDE.

###### Theorem 2 (Augmented forward SDE) .

Let ${\bm{x}}$ be a random process defined by equation 1. Then

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \log p_{t}({\bm{x}}_{t})\end{bmatrix}=\begin{bmatrix}f(t){\bm{x}}_{t}\\ F(t,{\bm{x}}_{t})\end{bmatrix}dt+g(t)\begin{bmatrix}{\bm{I}}_{D}\\ \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d{ \textnormal{W}}_{t},$ (5) $$

where

$$ $F(t,{\bm{x}}_{t})=-\operatorname{div}_{\bm{x}}\left(f(t){\bm{x}}_{t}-g^{2}(t) \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\right)+\frac{1}{2}g^ {2}(t)\|\definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{ rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\|^{2}.$ $$

Proof is similar to the reverse case and can be found in Appendix D.
The forward augmented SDE will prove useful for estimating the density of an arbitrary input point ${\bm{x}}$ (not necessarily sampled from the model).

Karras et al. () proposed a more general forward SDE than equation 1 for which we also derive the dynamics of $\log p_{t}({\bm{x}}_{t})$ in both directions (Appendix F).
Interestingly, we show that equation 1 is the only case for which the dynamics of $\log p_{t}({\bm{x}}_{t})$ in the reverse direction do not involve any higher order derivatives of $\log p_{t}({\bm{x}}_{t})$.

We can now track the likelihood for forward and reverse SDEs under the theoretical true $\nabla_{\bm{x}}\log p_{t}({\bm{x}})$.
However, under score approximation ${\bm{s}}(t,{\bm{x}})\approx\nabla_{\bm{x}}\log p_{t}({\bm{x}})$ the density solutions $\log p_{t}({\bm{x}}_{t})$ of Theorems 1 and 2 become biased, which we study next.

###### Theorem 1 (Augmented reverse SDE) .

###### Theorem 2 (Augmented forward SDE) .

### 2.3 Approximate reverse dynamics

We can substitute the approximate score ${\bm{s}}(t,{\bm{x}})\approx\nabla_{\bm{x}}\log p_{t}({\bm{x}})$ and assume $p_{T}^{\mathrm{ODE}}=p_{T}^{\mathrm{SDE}}=\mathcal{N}(\mathbf{0},\sigma_{T}^{2
}{\bm{I}}_{D})$.
The resulting SDE and ODE models are no longer equivalent, i.e. $p_{t}^{\mathrm{ODE}}\neq p_{t}^{\mathrm{SDE}}$ (Song et al., ; Lu et al., ).
The PF-ODE becomes

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \log p_{t}^{\mathrm{ODE}}({\bm{x}}_{t})\end{bmatrix}=\begin{bmatrix}f(t){\bm{x }}_{t}-\frac{1}{2}g^{2}(t)\definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor {tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{. }{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to 16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{29.16907pt}{16.84525pt}\pgfsys@curveto{31.52618pt}{16.84525 pt}{33.43695pt}{14.93448pt}{33.43695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{ 4.26788pt}\pgfsys@curveto{33.43695pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0.9}{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{29 .16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.42262pt}{32.01433pt}{14.14 877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt}{4.26788pt} \pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.16907pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\\ -f(t)D+\frac{1}{2}g^{2}(t)\definecolor{tcbcolback}{rgb}{1,0.85,0.85} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to23.66 pt{\vbox to15pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{10.72736pt}\pgfsys@curveto{0.0pt}{13.08447pt}{1.91077pt} {14.99524pt}{4.26788pt}{14.99524pt}\pgfsys@lineto{19.38907pt}{14.99524pt} \pgfsys@curveto{21.74619pt}{14.99524pt}{23.65695pt}{13.08447pt}{23.65695pt}{10 .72736pt}\pgfsys@lineto{23.65695pt}{4.26788pt}\pgfsys@curveto{23.65695pt}{1.91 077pt}{21.74619pt}{0.0pt}{19.38907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{10.7273 6pt}\pgfsys@curveto{1.42262pt}{12.29877pt}{2.69647pt}{13.57262pt}{4.26788pt}{1 3.57262pt}\pgfsys@lineto{19.38907pt}{13.57262pt}\pgfsys@curveto{20.96048pt}{13 .57262pt}{22.23433pt}{12.29877pt}{22.23433pt}{10.72736pt}\pgfsys@lineto{22.234 33pt}{4.26788pt}\pgfsys@curveto{22.23433pt}{2.69647pt}{20.96048pt}{1.42262pt}{ 19.38907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{4.62817pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\operatorname{div}_{\bm{x}}$}}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{} \hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss }}\lxSVG@closescope\endpgfpicture}}\definecolor{tcbcolback}{rgb}{0.9,0.9,1} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to33.44 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16.84525pt} \pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43695pt}{12 .57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.43695pt}{1.91 077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\end{bmatrix}dt,$ (6) $$

which can track its marginal log-density $\log p_{t}^{\mathrm{ODE}}({\bm{x}}_{t})$ exactly.
However, substituting the true score with ${\bm{s}}(t,{\bm{x}})$ in the augmented reverse SDE 4 incurs estimation error in $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$.
We characterise the error formally in a novel theorem:

###### Theorem 3 (Approximate Augmented Reverse SDE) .

Let ${\bm{s}}(t,{\bm{x}})$ be an approximation of the score function. Let ${\bm{x}}_{T}\sim p_{T}$ and define an auxiliary process $r$ starting at $r_{T}=\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T})$. If

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ r_{t}\end{bmatrix}=\begin{bmatrix}f(t){\bm{x}}_{t}-g^{2}(t)\definecolor{ tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{ tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{ 0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox {\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{ 0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\\ -f(t)D-\frac{1}{2}g^{2}(t)\|\definecolor{tcbcolback}{rgb}{0.9,0.9,1} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to33.44 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16.84525pt} \pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43695pt}{12 .57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.43695pt}{1.91 077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\|^{2}\end{bmatrix}dt+g(t)\begin{bmatrix}{\bm {I}}_{D}\\ \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d\overline{{\textnormal{W}}} _{t},$ (7) $$

then ${\bm{x}}_{0}\sim p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$ and

$$ $r_{0}=\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})+{\textnormal{X}},$ (8) $$

where X is a random variable such that the bias of $r_{0}$ is given by

$$ $\mathbb{E}{\textnormal{X}}=\frac{T}{2}\mathbb{E}_{t\sim\mathcal{U}(0,T),{\bm{x }}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})}\Big{[}g^{2}(t)\|\definecolor{ tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{ tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{ 0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox {\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{ 0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}-\definecolor{tcbcolback}{rgb}{1,0.91,0.82} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.45787pt}{16.84525pt} \pgfsys@curveto{51.81499pt}{16.84525pt}{53.72575pt}{14.93448pt}{53.72575pt}{12 .57736pt}\pgfsys@lineto{53.72575pt}{4.26788pt}\pgfsys@curveto{53.72575pt}{1.91 077pt}{51.81499pt}{0.0pt}{49.45787pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15 .42262pt}{52.30313pt}{14.14877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.303 13pt}{4.26788pt}\pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{ 49.45787pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$}}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope \endpgfpicture}}\|^{2}\Big{]}\geq 0.$ (9) $$

See Appendix G for the proof and the definition of X.
Intuitively, the true density evolution of $p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$ is induced by $d{\bm{x}}_{t}$ in equation 7, and the auxiliary variable $r_{t}$ does not follow it perfectly. Since the equation 9 has an intractable score, we seek more practical alternatives to measuring the accuracy of $r_{0}$ in the next Section.

###### Theorem 3 (Approximate Augmented Reverse SDE) .

## 3 Accuracy of the density estimation

We analyse the accuracy of the $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$ estimates in equation 8. We begin with the approximate forward dynamics which will provide a method for bounding the estimation error of $r_{0}$.

### 3.1 Approximate forward dynamics

In contrast to Theorem 3, when we replace the true score function with ${\bm{s}}$ in the forward direction, we underestimate $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$ on average.

###### Theorem 4 (Approximate Augmented Forward SDE) .

Let ${\bm{s}}(t,{\bm{x}}_{t})$ be the model approximating the score function and ${\bm{x}}_{0}\in\mathbb{R}^{D}$ given. Define an auxiliary process $\omega$ starting at $\omega_{0}=0$. If

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \omega_{t}\end{bmatrix}=\begin{bmatrix}f(t){\bm{x}}_{t}\\ -f(t)D+g^{2}(t)\Big{(}\frac{1}{2}\|\definecolor{tcbcolback}{rgb}{0.9,0.9,1} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to33.44 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16.84525pt} \pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43695pt}{12 .57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.43695pt}{1.91 077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\|^{2}+\definecolor{tcbcolback}{rgb}{ 1,0.85,0.85}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{ rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to23.66pt{\vbox to15pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0 .0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{10.72736pt}\pgfsys@curveto{0.0pt}{13.08447pt}{1.91077pt} {14.99524pt}{4.26788pt}{14.99524pt}\pgfsys@lineto{19.38907pt}{14.99524pt} \pgfsys@curveto{21.74619pt}{14.99524pt}{23.65695pt}{13.08447pt}{23.65695pt}{10 .72736pt}\pgfsys@lineto{23.65695pt}{4.26788pt}\pgfsys@curveto{23.65695pt}{1.91 077pt}{21.74619pt}{0.0pt}{19.38907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{10.7273 6pt}\pgfsys@curveto{1.42262pt}{12.29877pt}{2.69647pt}{13.57262pt}{4.26788pt}{1 3.57262pt}\pgfsys@lineto{19.38907pt}{13.57262pt}\pgfsys@curveto{20.96048pt}{13 .57262pt}{22.23433pt}{12.29877pt}{22.23433pt}{10.72736pt}\pgfsys@lineto{22.234 33pt}{4.26788pt}\pgfsys@curveto{22.23433pt}{2.69647pt}{20.96048pt}{1.42262pt}{ 19.38907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{4.62817pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\operatorname{div}_{\bm{x}}$}}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{} \hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss }}\lxSVG@closescope\endpgfpicture}}\definecolor{tcbcolback}{rgb}{0.9,0.9,1} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to33.44 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16.84525pt} \pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43695pt}{12 .57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.43695pt}{1.91 077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\Big{)}\end{bmatrix}dt+g(t)\begin{bmatrix}{ \bm{I}}_{D}\\ \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d{\textnormal{W}}_{t}.$ (10) $$

Then

$$ $\omega_{T}=\log p_{T}^{\mathrm{SDE}}({\bm{x}}_{T})-\log p_{0}^{\mathrm{SDE}}({ \bm{x}}_{0})+{\textnormal{Y}}_{{\bm{x}}_{0}},$ (11) $$

where ${\textnormal{Y}}_{{\bm{x}}_{0}}$ is a random variable such that

$$ $\mathbb{E}{\textnormal{Y}}_{{\bm{x}}_{0}}=\frac{T}{2}\mathbb{E}_{t\sim\mathcal {U}(0,T)}\mathbb{E}_{{\bm{x}}_{t}\sim p({\bm{x}}_{t}|{\bm{x}}_{0})}g^{2}(t)\| \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}-\definecolor{tcbcolback}{rgb}{1,0.91,0.82} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.45787pt}{16.84525pt} \pgfsys@curveto{51.81499pt}{16.84525pt}{53.72575pt}{14.93448pt}{53.72575pt}{12 .57736pt}\pgfsys@lineto{53.72575pt}{4.26788pt}\pgfsys@curveto{53.72575pt}{1.91 077pt}{51.81499pt}{0.0pt}{49.45787pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15 .42262pt}{52.30313pt}{14.14877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.303 13pt}{4.26788pt}\pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{ 49.45787pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$}}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope \endpgfpicture}}\|^{2}\geq 0.$ (12) $$

See Appendix G for the proof and the definition of ${\textnormal{Y}}_{{\bm{x}}_{0}}$. Due to the drift $\operatorname{div}$ operator, the evaluation of $d\omega_{t}$ is computationally comparable to $d\log p^{\mathrm{ODE}}_{t}({\bm{x}}_{t})$.
Interestingly, Theorem 4 completes a known lower bound into a novel identity,

$$ $\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})=\underbrace{\mathbb{E}{\textnormal{Y}} _{{\bm{x}}_{0}}}_{\geq 0}\underbrace{-\frac{e^{\lambda_{\text{min}}}}{2}\|{\bm {x}}_{0}\|^{2}+\frac{T}{2}\mathbb{E}_{t,{\bm{\varepsilon}}}\bigg{[}-\frac{d \lambda_{t}}{dt}\big{\|}\sigma_{t}\definecolor{tcbcolback}{rgb}{0.9,0.9,1} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to66.7 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{62.43222pt}{16.84525pt} \pgfsys@curveto{64.78934pt}{16.84525pt}{66.7001pt}{14.93448pt}{66.7001pt}{12.5 7736pt}\pgfsys@lineto{66.7001pt}{4.26788pt}\pgfsys@curveto{66.7001pt}{1.91077 pt}{64.78934pt}{0.0pt}{62.43222pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{62.43222pt}{15.42262pt}\pgfsys@curveto{64.00363pt}{15.422 62pt}{65.27748pt}{14.14877pt}{65.27748pt}{12.57736pt}\pgfsys@lineto{65.27748pt }{4.26788pt}\pgfsys@curveto{65.27748pt}{2.69647pt}{64.00363pt}{1.42262pt}{62.4 3222pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,\alpha_{t}{\bm{x}}_{0}+\sigma_{t}{\bm{\varepsilon}})$ }}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope \endpgfpicture}}+{\bm{\varepsilon}}\big{\|}^{2}\bigg{]}+C}_{\mathrm{ELBO}({\bm {x}}_{0})\text{ \cite[citep]{(\@@bibref{AuthorsPhrase1Year}{song2021maximum, kingma2021variational}{\@@citephrase{, }}{})}}},$ (13) $$

where $t\sim\mathcal{U}(0,T)$, ${\bm{\varepsilon}}\sim\mathcal{N}(\mathbf{0},{\bm{I}}_{D})$ and $C=-\frac{D}{2}\left(1+\log(2\pi\sigma_{0}^{2})\right)$ (see 1).
$\mathrm{ELBO}({\bm{x}})$ is the standard tool for estimating the SDE’s model likelihood of an arbitrary point ${\bm{x}}$.

###### Theorem 4 (Approximate Augmented Forward SDE) .

### 3.2 Estimation bias of log ⁡ p 0 SDE superscript subscript 𝑝 0 SDE \log p_{0}^{\mathrm{SDE}} roman_log italic_p start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_SDE end_POSTSUPERSCRIPT

Figure: Figure 4: $r_{0}>\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})>\mathrm{ELBO}({\bm{x}}_{0})$ correlate strongly.
Refer to caption: extracted/5973435/figures/kl-ro-ode-cifar-both-models.png

We are interested in using the stochastic sampler to obtain high-quality sample-density pairs $({\bm{x}}_{0},\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{0}))$.
We have now discussed two ways of estimating $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$: with $r_{0}$ (equation 7) and $\mathrm{ELBO}({\bm{x}}_{0})$ (equation 13).
Their estimation errors $\varepsilon_{r}(r_{0},{\bm{x}}_{0})=r_{0}-\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{
0})$ and $\varepsilon_{\mathrm{ELBO}}({\bm{x}}_{0})=\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{
0})-\mathrm{ELBO}({\bm{x}}_{0})$ are intractable to estimate due to the presence of unknown score $\nabla\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$.

However, we can provide an upper bound on the bias of both estimators,

$$ $\underbrace{\mathbb{E}_{{\bm{x}}_{0}\sim p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})} \left[\varepsilon_{\mathrm{ELBO}}({\bm{x}}_{0})\right]}_{\geq 0\>(\text{ equation~{}\ref{eq:fwd-bias}})}+\underbrace{\mathbb{E}_{({\bm{x}}_{0},r_{0})} \left[\varepsilon_{r}(r_{0},{\bm{x}}_{0})\right]}_{\geq 0\>(\text{equation~{} \ref{eq:X}})}=\underbrace{\mathbb{E}_{({\bm{x}}_{0},r_{0})}\Big{[}r_{0}- \mathrm{ELBO}({\bm{x}}_{0})\Big{]}.}_{\mathcal{R}({\bm{s}})(\text{tractable})}$ (14) $$

We can thus sample $({\bm{x}}_{0},r_{0})$ using equation 7 and the average difference $r_{0}-\mathrm{ELBO}({\bm{x}}_{0})$ gives an upper bound on the bias of both $r_{0}$ and $\mathrm{ELBO}({\bm{x}}_{0})$.
This can be useful to assess the accuracy of both $r_{0}$ and $\mathrm{ELBO}({\bm{x}}_{0})$ as density estimates for stochastic samples ${\bm{x}}_{0}$.
Furthermore, we can estimate how much $p_{0}^{\mathrm{{SDE}}}$ differs from $p_{0}^{\mathrm{{ODE}}}$ by providing bounds for $\operatorname{KL}[p^{\mathrm{SDE}}_{0}||p^{\mathrm{ODE}}_{0}]$

$$ $\underbrace{\mathbb{E}_{{\bm{x}}_{0}\sim p^{\mathrm{SDE}}({\bm{x}}_{0})}\Big{[ }\mathrm{ELBO}({\bm{x}}_{0})-\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})\Big{]}}_{ \mathcal{R}^{L}({\bm{s}})\text{ (tractable)}}\leq\operatorname{KL}\Big{[}p_{0} ^{\mathrm{SDE}}||p_{0}^{\mathrm{ODE}}\Big{]}\leq\underbrace{\mathbb{E}_{({\bm{ x}}_{0},r_{0})}\Big{[}r_{0}-\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})\Big{]}}_{ \mathcal{R}^{U}({\bm{s}})\text{ (tractable)}}.$ (15) $$

$\mathcal{R}^{U}$ and $\mathcal{R}^{L}$ are novel practical tools for measuring the difference between $p_{0}^{\mathrm{SDE}}$ and $p_{0}^{\mathrm{ODE}}$.
As a demonstration, we train two versions of a diffusion model on CIFAR-10 (Krizhevsky et al., ), one with maximum likelihood training (Kingma et al., ; Song et al., ) and one optimized for sample quality(^1^11Both models are trained with the weighted ELBO objective. The one optimized for likelihood uses $w(\lambda)=1$ (unweighted) and the one optimized for sample quality uses $w(\lambda)=\mathrm{sigmoid}(-\lambda+2)$.) (Kingma & Gao, ).
We then generate 512 samples of $({\bm{x}}_{0},r_{0})$ with equation 7 and for each ${\bm{x}}_{0}$ we estimated $\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})$ using equation 6 and $\mathrm{ELBO}({\bm{x}}_{0})$ using equation 13.

For both models we found very high correlations between $r_{0}$ and $\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})$ at $0.996$ and $0.999$.
Surprisingly, it is the model optimized for sample quality that yielded both a higher correlation and lower values of $\mathcal{R}^{U}({\bm{s}})$ and $\mathcal{R}({\bm{s}})$ suggesting a smaller difference between $p_{0}^{\mathrm{SDE}}$ and $p_{0}^{\mathrm{ODE}}$ and a lower bias of $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$ estimation (Figure 4).
We also estimated $\mathcal{R}^{L}({\bm{s}})$, which was negative for both models which is a trivial lower bound for $\operatorname{KL}[p_{0}^{\mathrm{SDE}}||p_{0}^{\mathrm{ODE}}]\geq 0$.

## 4 Mode estimation

Figure: Figure 5: Equation 16 accurately recovers the mode-tracking curve.
Refer to caption: extracted/5973435/figures/mode-tracking-curve-updated.png

Many diffusion models are trained by implicitly maximizing weighted ELBO (Kingma & Gao, ), and can be interpreted as likelihood-based models.
Karras et al. () emphasized the role of likelihood by explaining the empirical success of guided diffusion models by their ability to limit low density samples $p_{0}({\bm{x}}_{0})$.
We explore this idea further by asking:
*what if we aim for samples with the highest $p_{0}({\bm{x}}_{0})$?*

### 4.1 Mode-tracking curve

We first go back to assuming a perfectly estimated score function $\nabla_{\bm{x}}\log p_{t}({\bm{x}})$. Let $p({\bm{x}}_{0}|{\bm{x}}_{t})$ denote the denoising distribution given by solving the reverse SDE (equation 2) from ${\bm{x}}_{t}$ to ${\bm{x}}_{0}$.
If we knew how to estimate the denoising mode ${\bm{y}}_{0}({\bm{x}}_{t})=\operatorname*{arg\,max}p({\bm{x}}_{0}|{\bm{x}}_{t})$, we could bias the sampler towards higher likelihood regions by first taking a regular noisy samples ${\bm{x}}_{t}\sim p_{t}({\bm{x}}_{t})$ at various times $t$, and pushing them to deterministic modes ${\bm{y}}_{0}({\bm{x}}_{t})$.

We approach this by asking a seemingly more difficult question: can we find a mode-tracking curve, i.e. ${\bm{y}}_{s}$ such that $p({\bm{y}}_{s}|{\bm{x}}_{t})=\max_{{\bm{x}}_{s}}p({\bm{x}}_{s}|{\bm{x}}_{t})$ for all $s<t$ (See Figure 5)?
We show that whenever such a smooth curve exists it is given by an ODE:

###### Theorem 5 (Mode-tracking ODE) .

Let $t\in(0,T]$ and ${\bm{x}}_{t}\in\mathbb{R}^{D}$ a noisy sample. If there exists a smooth curve $s\mapsto{\bm{y}}_{s}$ such that $p({\bm{y}}_{s}|{\bm{x}}_{t})=\max_{{\bm{x}}_{s}}p({\bm{x}}_{s}|{\bm{x}}_{t})$, then ${\bm{y}}_{t}={\bm{x}}_{t}$ and for $s<t$

$$ $\frac{d}{ds}{\bm{y}}_{s}=f(s){\bm{y}}_{s}-g^{2}(s)\definecolor{tcbcolback}{rgb }{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin} {rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to54.22pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt \lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.95567pt}{16 .84525pt}\pgfsys@curveto{52.31279pt}{16.84525pt}{54.22356pt}{14.93448pt}{54.22 356pt}{12.57736pt}\pgfsys@lineto{54.22356pt}{4.26788pt}\pgfsys@curveto{54.2235 6pt}{1.91077pt}{52.31279pt}{0.0pt}{49.95567pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.95567pt}{15.42262pt}\pgfsys@curveto{51.52708pt}{15 .42262pt}{52.80093pt}{14.14877pt}{52.80093pt}{12.57736pt}\pgfsys@lineto{52.800 93pt}{4.26788pt}\pgfsys@curveto{52.80093pt}{2.69647pt}{51.52708pt}{1.42262pt}{ 49.95567pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\frac{1}{2}g^{2}(s) \definecolor{tcbcolback}{rgb}{1,0.85,0.85}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to42.14pt{\vbox to18.65pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{14.3818pt} \pgfsys@curveto{0.0pt}{16.73892pt}{1.91077pt}{18.64969pt}{4.26788pt}{18.64969 pt}\pgfsys@lineto{37.87372pt}{18.64969pt}\pgfsys@curveto{40.23083pt}{18.64969 pt}{42.1416pt}{16.73892pt}{42.1416pt}{14.3818pt}\pgfsys@lineto{42.1416pt}{4.26 788pt}\pgfsys@curveto{42.1416pt}{1.91077pt}{40.23083pt}{0.0pt}{37.87372pt}{0.0 pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1. 91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0.85}{0.85}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{14.3818pt}\pgfsys@curveto{1.42262 pt}{15.95322pt}{2.69647pt}{17.22707pt}{4.26788pt}{17.22707pt}\pgfsys@lineto{37 .87372pt}{17.22707pt}\pgfsys@curveto{39.44513pt}{17.22707pt}{40.71898pt}{15.95 322pt}{40.71898pt}{14.3818pt}\pgfsys@lineto{40.71898pt}{4.26788pt} \pgfsys@curveto{40.71898pt}{2.69647pt}{39.44513pt}{1.42262pt}{37.87372pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{A}}(s,{\bm{y}}_{s})^{-1}$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\definecolor{ tcbcolback}{rgb}{1,0.85,0.85}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor {tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{ 0,0,0}\leavevmode\hbox to65.5pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{ \hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{ 0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{61.23552pt}{16 .84525pt}\pgfsys@curveto{63.59264pt}{16.84525pt}{65.5034pt}{14.93448pt}{65.503 4pt}{12.57736pt}\pgfsys@lineto{65.5034pt}{4.26788pt}\pgfsys@curveto{65.5034pt} {1.91077pt}{63.59264pt}{0.0pt}{61.23552pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0 pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{61.23552pt}{15.42262pt}\pgfsys@curveto{62.80693pt}{15 .42262pt}{64.08078pt}{14.14877pt}{64.08078pt}{12.57736pt}\pgfsys@lineto{64.080 78pt}{4.26788pt}\pgfsys@curveto{64.08078pt}{2.69647pt}{62.80693pt}{1.42262pt}{ 61.23552pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{{\bm{y}}}\Delta_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})$}}}} \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope \endpgfpicture}},$ (16) $$

where ${\bm{A}}(s,{\bm{y}})=\left(\nabla_{\bm{y}}^{2}\log p_{s}({\bm{y}})-\psi(s){\bm
{I}}_{D}\right)$, $\psi(s)=\frac{1}{\sigma_{s}^{2}}\frac{e^{\lambda_{t}}}{e^{\lambda_{s}}-e^{
\lambda_{t}}}$, and $\Delta_{{\bm{y}}}=\sum_{i}\frac{\partial^{2}}{\partial y_{i}^{2}}$ is the Laplace operator. In particular:

$$ $p({\bm{y}}_{0}|{\bm{x}}_{t})=\max_{{\bm{x}}_{0}}p({\bm{x}}_{0}|{\bm{x}}_{t}).$ (17) $$

The proof and the statement without assuming invertibility of ${\bm{A}}$ can be found in Appendix H. We visualize on a mixture of 1D Gaussians in Figure 5 how the solution of the mode tracking ODE (equation 16) correctly recovers the denoising mode curve, i.e. the mode of $p({\bm{x}}_{s}|{\bm{x}}_{t})$ for all $s<t$.

Figure: Figure 6: Equation 16 enables controlling the likelihood-diversity tradeoff. Blue dots are samples from the data distribution, i.e. a mixture of four Gaussians with the bottom left component having the highest weight. Orange samples are generated with Algorithm 1.
Refer to caption: extracted/5973435/figures/outlier_elimination.png

###### Theorem 5 (Mode-tracking ODE) .

### 4.2 High-probability seeking ODE

Using Theorem 5 in high-dimensional data is problematic. A smooth mode-tracking curve needs to exist, which is difficult to verify and need not hold (Appendix J).
Moreover, equation 16 requires computing and inverting the Hessian and estimating third-order derivatives, which is costly (Meng et al., ).
We propose an approximation of equation 16 by noting its high-order terms disappear under Gaussian data.

###### Remark 1 (High-probability ODE or HP-ODE) .

If $p_{0}$ is Gaussian, then equation 16 becomes

$$ $d{\bm{y}}_{s}=\left(f(s){\bm{y}}_{s}-g^{2}(s)\definecolor{tcbcolback}{rgb}{ 1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{ rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to54.48pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt \lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{50.20973pt}{16 .84525pt}\pgfsys@curveto{52.56685pt}{16.84525pt}{54.47762pt}{14.93448pt}{54.47 762pt}{12.57736pt}\pgfsys@lineto{54.47762pt}{4.26788pt}\pgfsys@curveto{54.4776 2pt}{1.91077pt}{52.56685pt}{0.0pt}{50.20973pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{50.20973pt}{15.42262pt}\pgfsys@curveto{51.78114pt}{15 .42262pt}{53.055pt}{14.14877pt}{53.055pt}{12.57736pt}\pgfsys@lineto{53.055pt}{ 4.26788pt}\pgfsys@curveto{53.055pt}{2.69647pt}{51.78114pt}{1.42262pt}{50.20973 pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1 .42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{s}({\bm{y}}_{s})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\right)ds,$ (18) $$

i.e. the drift term of reverse SDE equation 2. If ${\bm{y}}_{t}={\bm{x}}_{t}$ and equation 18 holds for $s<t$, then

$$ ${\bm{y}}_{0}=\operatorname*{arg\,max}_{{\bm{x}}_{0}}p({\bm{x}}_{0}|{\bm{x}}_{t })+\mathcal{O}(e^{-\lambda_{\mathrm{max}}}).$ (19) $$

See Appendix I for the proof.
Even though equation 18 finds the mode of $p({\bm{x}}_{0}|{\bm{x}}_{t})$ only in the Gaussian case (for sufficiently small $e^{-\lambda_{\mathrm{max}}}$), we will empirically show that even for non-Gaussian data it finds points with much higher likelihoods than regular samples.

Figure: Algorithm 1 High probability sampling

In Algorithm 1 we propose a novel high-probability sampler that uses equation 18 to bias the sampling towards higher likelihood regions. We choose a threshold time $t$ and sample ${\bm{x}}_{t}\sim p_{t}({\bm{x}}_{t})$, and then estimate ${\bm{y}}_{0}({\bm{x}}_{t})$ by solving equation 18 from $s=t$ to $s=0$. Figure 6 shows how the threshold controls the tradeoff between likelihood and diversity in a toy mixture of Gaussians.

###### Remark 1 (High-probability ODE or HP-ODE) .

### 4.3 Estimating mode densities for Real-world data

Figure: Figure 7: Algorithm 1 generates images with higher likelihoods than regular samples. For different noise levels $t$, compare the high probability ${\bm{y}}_{0}$ (equation 18) with random samples ${\bm{x}}_{0}$ on CIFAR-10. We find that the percentage of samples with higher likelihoods than ${\bm{y}}_{0}$ is 0 0.
Refer to caption: extracted/5973435/figures/updated-cifar-percentage-figure-high-prob.png

We will next discuss how to evaluate the density of the high-probability samples ${\bm{y}}_{0}|{\bm{x}}_{t}$ and regular samples ${\bm{x}}_{0}|{\bm{x}}_{t}$ to empirically show that $p({\bm{y}}_{0}|{\bm{x}}_{t})>p({\bm{x}}_{0}|{\bm{x}}_{t})$ for ${\bm{x}}_{0}\sim p({\bm{x}}_{0}|{\bm{x}}_{t})$. For any ${\bm{x}}_{0}$, the denoising likelihood can be decomposed using Bayes’ rule

$$ $\log p_{0|t}({\bm{x}}_{0}|{\bm{x}}_{t})=\underbrace{\log p_{t|0}({\bm{x}}_{t}| {\bm{x}}_{0})}_{\mathcal{N}({\bm{x}}_{t}|\alpha_{t}{\bm{x}}_{0},\sigma_{t}^{2} {\bm{I}}_{D})}+\log p_{0}({\bm{x}}_{0})-\log p_{t}({\bm{x}}_{t}).$ (20) $$

For ${\bm{x}}_{0}\sim p({\bm{x}}_{0}|{\bm{x}}_{t})$ we can use equation 4 from $({\bm{x}}_{t},r_{t}=0)$ to obtain $r_{0}=\log p_{0}({\bm{x}}_{0})-\log p_{t}({\bm{x}}_{t})$.
To estimate $\log p({\bm{y}}_{0}|{\bm{x}}_{t})$ we could use the PF-ODE, but that is inefficient.(^2^22On top of solving HP-ODE we would need to solve the augmented PF-ODE twice: once for ${\bm{y}}_{t}$ from $t$ to $T$ and once for ${\bm{y}}_{0}$ from 0 0 to $T$. Instead, we perform one augmented ((18) + (22)) HP-ODE solve from $t$ to 0 0.)
Instead, we can obtain $d\log p_{s}({\bm{y}}_{s})$ under HP-ODE with a convenient, and to our knowledge novel, lemma:

###### Lemma 1 (General instantaneous change of variables) .

Consider a CNF given by
$d{\bm{x}}_{t}=f_{1}(t,{\bm{x}}_{t})dt$
with prior $p_{T}$ and marginal distributions $p_{t}$, and a particle following some different dynamical system
$d{\bm{z}}_{t}=f_{2}(t,{\bm{z}}_{t})dt$. Then, if $f_{1}$ and $f_{2}$ are uniformly Lipschitz in the second argument and continuous in the first, we have:

$$ $\frac{d\log p_{t}({\bm{z}}_{t})}{dt}=-\definecolor{tcbcolback}{rgb}{ 1,0.85,0.85}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{ rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to23.31pt{\vbox to15pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0 .0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{10.72736pt}\pgfsys@curveto{0.0pt}{13.08447pt}{1.91077pt} {14.99524pt}{4.26788pt}{14.99524pt}\pgfsys@lineto{19.03908pt}{14.99524pt} \pgfsys@curveto{21.3962pt}{14.99524pt}{23.30696pt}{13.08447pt}{23.30696pt}{10. 72736pt}\pgfsys@lineto{23.30696pt}{4.26788pt}\pgfsys@curveto{23.30696pt}{1.910 77pt}{21.3962pt}{0.0pt}{19.03908pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{10.7273 6pt}\pgfsys@curveto{1.42262pt}{12.29877pt}{2.69647pt}{13.57262pt}{4.26788pt}{1 3.57262pt}\pgfsys@lineto{19.03908pt}{13.57262pt}\pgfsys@curveto{20.61049pt}{13 .57262pt}{21.88434pt}{12.29877pt}{21.88434pt}{10.72736pt}\pgfsys@lineto{21.884 34pt}{4.26788pt}\pgfsys@curveto{21.88434pt}{2.69647pt}{20.61049pt}{1.42262pt}{ 19.03908pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{4.62817pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\operatorname{div}_{\bm{z}}$}}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{} \hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss }}\lxSVG@closescope\endpgfpicture}}f_{1}(t,{\bm{z}}_{t})+\big{(}f_{2}(t,{\bm{z }}_{t})-f_{1}(t,{\bm{z}}_{t})\big{)}^{T}\definecolor{tcbcolback}{rgb}{ 1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{ rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to52.75pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt \lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{48.48291pt}{16 .84525pt}\pgfsys@curveto{50.84003pt}{16.84525pt}{52.7508pt}{14.93448pt}{52.750 8pt}{12.57736pt}\pgfsys@lineto{52.7508pt}{4.26788pt}\pgfsys@curveto{52.7508pt} {1.91077pt}{50.84003pt}{0.0pt}{48.48291pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0 pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{48.48291pt}{15.42262pt}\pgfsys@curveto{50.05432pt}{15 .42262pt}{51.32817pt}{14.14877pt}{51.32817pt}{12.57736pt}\pgfsys@lineto{51.328 17pt}{4.26788pt}\pgfsys@curveto{51.32817pt}{2.69647pt}{50.05432pt}{1.42262pt}{ 48.48291pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{z}}\log p_{t}({\bm{z}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}.$ (21) $$

The proof is in Appendix K. When $f_{1}=f_{2}$ we recover the standard formula Chen et al. (). By setting $f_{1}=$PF-ODE (3) and $f_{2}=$HP-ODE (18) we augment the HP-ODE with its density evolution

$$ $\frac{d\log p_{s}({\bm{y}}_{s})}{ds}=-f(s)D+\frac{1}{2}g^{2}(s)\definecolor{ tcbcolback}{rgb}{1,0.85,0.85}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor {tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{ 0,0,0}\leavevmode\hbox to23.4pt{\vbox to16.08pt{\pgfpicture\makeatletter\hbox{ \hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{ 0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{11.81624pt}\pgfsys@curveto{0.0pt}{14.17336 pt}{1.91077pt}{16.08412pt}{4.26788pt}{16.08412pt}\pgfsys@lineto{19.13501pt}{16 .08412pt}\pgfsys@curveto{21.49213pt}{16.08412pt}{23.4029pt}{14.17336pt}{23.402 9pt}{11.81624pt}\pgfsys@lineto{23.4029pt}{4.26788pt}\pgfsys@curveto{23.4029pt} {1.91077pt}{21.49213pt}{0.0pt}{19.13501pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0 pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{11.8162 4pt}\pgfsys@curveto{1.42262pt}{13.38765pt}{2.69647pt}{14.6615pt}{4.26788pt}{14 .6615pt}\pgfsys@lineto{19.13501pt}{14.6615pt}\pgfsys@curveto{20.70642pt}{14.66 15pt}{21.98027pt}{13.38765pt}{21.98027pt}{11.81624pt}\pgfsys@lineto{21.98027pt }{4.26788pt}\pgfsys@curveto{21.98027pt}{2.69647pt}{20.70642pt}{1.42262pt}{19.1 3501pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.71706pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\operatorname{div}_{\bm{y}}$}}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{} \hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss }}\lxSVG@closescope\endpgfpicture}}\definecolor{tcbcolback}{rgb}{1,0.91,0.82} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to54.22 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{49.95567pt}{16.84525pt} \pgfsys@curveto{52.31279pt}{16.84525pt}{54.22356pt}{14.93448pt}{54.22356pt}{12 .57736pt}\pgfsys@lineto{54.22356pt}{4.26788pt}\pgfsys@curveto{54.22356pt}{1.91 077pt}{52.31279pt}{0.0pt}{49.95567pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0 .91}{0.82}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.5773 6pt}\pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{1 5.42262pt}\pgfsys@lineto{49.95567pt}{15.42262pt}\pgfsys@curveto{51.52708pt}{15 .42262pt}{52.80093pt}{14.14877pt}{52.80093pt}{12.57736pt}\pgfsys@lineto{52.800 93pt}{4.26788pt}\pgfsys@curveto{52.80093pt}{2.69647pt}{51.52708pt}{1.42262pt}{ 49.95567pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{y}}\log p_{s}({\bm{y}}_{s})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}-\frac{1}{2}g^{2}(s) \big{\|}\definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{ rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to54.22pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.95567pt}{16.84525pt}\pgfsys@curveto{52.31279pt}{16.84525 pt}{54.22356pt}{14.93448pt}{54.22356pt}{12.57736pt}\pgfsys@lineto{54.22356pt}{ 4.26788pt}\pgfsys@curveto{54.22356pt}{1.91077pt}{52.31279pt}{0.0pt}{49.95567pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .95567pt}{15.42262pt}\pgfsys@curveto{51.52708pt}{15.42262pt}{52.80093pt}{14.14 877pt}{52.80093pt}{12.57736pt}\pgfsys@lineto{52.80093pt}{4.26788pt} \pgfsys@curveto{52.80093pt}{2.69647pt}{51.52708pt}{1.42262pt}{49.95567pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{y}}\log p_{s}({\bm{y}}_{s})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\big{\|}^{2}.$ (22) $$

We trained a diffusion model on CIFAR-10 and generated ${\bm{x}}_{t}$ for different noise levels $t$ and compared the high-density samples ${\bm{y}}_{0}|{\bm{x}}_{t}$ (Algorithm 1) against regular samples ${\bm{x}}_{0}\sim p({\bm{x}}_{0}|{\bm{x}}_{t})$.
We found that $p({\bm{y}}_{0}|{\bm{x}}_{t})>p({\bm{x}}_{0}|{\bm{x}}_{t})$ for all samples across different noise levels $t$ (See Figure 7).

###### Lemma 1 (General instantaneous change of variables) .

## 5 High-resolution diffusion probability landscape

Figure: Figure 8: Density estimates correlate with information content (.png size)
Refer to caption: extracted/5973435/figures/nll-vs-png-ffhq.png

We demonstrated that algorithm 1 can generate images with much higher likelihoods than regular samples.
However, after visually inspecting the high density samples ${\bm{y}}_{0}$ in Figure 7 we found that they correspond to blurry images with much less detail than regular samples.

To gain more insight we perform high-probability sampling (algorithm 1) on higher-resolution data with pre-trained diffusion models.(^3^33We used FFHQ-256 and Churches-256 models from [github.com/yang-song/score_sde_pytorch](github.com/yang-song/score_sde_pytorch) and ImageNet-64 from [github.com/NVlabs/edm](github.com/NVlabs/edm))
We tested different values of the threshold hyperparameter $t$ and found that for multiple models and datasets, the generated samples had higher average likelihood than regular samples. Moreover, we found that the samples with the highest likelihood were blurry images.
Surprisingly, we found that for higher values of $t$, the samples were unnatural cartoons, but still received higher likelihoods than regular samples. The training datasets do not contain any cartoon images. See Figure 9.

Figure: Figure 9: Diffusion models assign highest probabilities to unrealistic images. Left: High-probability samples generated with Algorithm 1 for various values of the threshold parameter $t$. Right: distributions of negative log-likelihood (in bits-per-dim). Across different models and datasets, Algorithm 1 finds unnatural images, which have higher densities than regular samples.
Refer to caption: extracted/5973435/figures/high-resolution-high-p-sampling-all-models.png

Figure: Figure 10: Blurring increases likelihood. Left: Two FFHQ-256 images with different amounts of blur and corresponding negative loglikelihoods (NLL). Right: Distributions of NLL for different amounts of added blur.
Refer to caption: extracted/5973435/figures/ffhq-blurring-combined.png

Inspired by the empirical observation that the highest likelihood generated samples are blurry images we performed the following two experiments.
First, we add different amounts of blur to FFHQ-256 test images and measure the likelihood of the distorted image.
We found that blurring always increases the likelihood and that the increase is proportional to the strength of blurring (Figure 10).

Second, we compare the model’s likelihood estimation with the image’s file size after PNG compression.
The smaller the PNG file size, the less information in the image.
For FFHQ-256, we found a 97% correlation between the model’s estimates of $\log p_{0}({\bm{x}}_{0})$ and the amount of information contained in an image (Figure 8).

## 6 Discussion

#### Variance of log ⁡ p 0 SDE ⁢ ( 𝒙 0 ) superscript subscript 𝑝 0 SDE subscript 𝒙 0 \log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0}) roman_log italic_p start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_SDE end_POSTSUPERSCRIPT ( bold_italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) estimate.

In section 3 we discussed the accuracy of $r_{0}$, our novel estimate of $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$. Specifically, we provided tools to estimate the bound of its bias for stochastically generated samples. Based on the empirically measured correlation between $r_{0}$ and $\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})$ at over $0.99$, we hypothesize that the variance of $r_{0}-\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})$ is low whenever ${\bm{s}}(t,{\bm{x}})\approx\nabla\log p_{t}^{\mathrm{SDE}}({\bm{x}})$. However, we do not provide theoretical guarantees, or empirical estimates.

#### p 0 SDE superscript subscript 𝑝 0 SDE p_{0}^{\mathrm{SDE}} italic_p start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_SDE end_POSTSUPERSCRIPT vs p 0 ODE superscript subscript 𝑝 0 ODE p_{0}^{\mathrm{ODE}} italic_p start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_ODE end_POSTSUPERSCRIPT .

In equation 15 we show that $\mathcal{R}^{U}({\bm{s}})=\mathbb{E}_{({\bm{x}}_{0},r_{0})}[r_{0}-\log p_{0}^{
\mathrm{ODE}}({\bm{x}}_{0})]$ is an upper bound on $\operatorname{KL}[p^{\mathrm{SDE}}_{0}||p^{\mathrm{ODE}}_{0}]\geq 0$.
In particular, for $({\bm{x}}_{0},r_{0})$ sampled with equation 7 we must have *on average* $r_{0}\geq\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})$. However, for two different models, we found that $r_{0}\geq\log p_{0}^{\mathrm{ODE}}({\bm{x}}_{0})$ holds *for every sample*. We hypothesize this is a more widespread phenomenon, but do not prove it.

#### Towards exact estimates of log ⁡ p 0 SDE ⁢ ( 𝒙 0 ) superscript subscript 𝑝 0 SDE subscript 𝒙 0 \log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0}) roman_log italic_p start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT roman_SDE end_POSTSUPERSCRIPT ( bold_italic_x start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) .

We showed in equation 9 that the bias of $r_{0}$ is given by $\mathbb{E}(r_{0}-\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0}))\propto\mathbb{E}_{t,
{\bm{x}}_{t}}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p_{t}^{
\mathrm{SDE}}({\bm{x}}_{t})\|^{2}$ for ${\bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$.
Similarly, equation 12 shows that $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})-\mathrm{ELBO}({\bm{x}}_{0})\propto
\mathbb{E}_{t,{\bm{x}}_{t}}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}
\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^{2}$ for ${\bm{x}}_{t}\sim p({\bm{x}}_{t}|{\bm{x}}_{0})$.
Both these errors could then be reduced to zero if ${\bm{s}}(t,{\bm{x}})=\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})$ for all $t,{\bm{x}}$.
However, for SDEs with linear drift (equation 1), this can only happen if $p_{t}^{\mathrm{SDE}}$ is Gaussian for all $t$ (Proposition B.1 in Lu et al. ()).
This is because an SDE with linear drift cannot transform a non-Gaussian $p_{0}$ into a Gaussian in finite time $T$.

To unlock exact likelihood estimation in diffusion SDEs, non-linear drift is necessary, such as the one proposed in Bartosh et al. ().
There it is possible to have $p_{T}$ Gaussian for finite $T$ and ${\bm{s}}(t,{\bm{x}})=\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})$ for all $t,{\bm{x}}$, in which case both $r_{0}$ and $\mathrm{ELBO}({\bm{x}}_{0})$ become exact (Theorem 7 and 3).

## 7 Related Work

We reference most of the related work in the main sections.

#### Likelihood estimation for diffusion models.

As we discussed in subsection 2.3 there is a distinction between diffusion ODEs and SDEs. For diffusion SDEs only lower bounds on likelihood are reported (Ho et al., ; Vahdat et al., ; Nichol & Dhariwal, ; Huang et al., ; Kingma et al., ; Kim et al., ). Exact likelihoods, on the other hand, are available for diffusion ODEs (Song et al., ; ; ; Dockhorn et al., ) and some works explicitly optimize for ODE likelihood (Lu et al., ; Zheng et al., ; Lai et al., ). For a comprehensive survey, we refer the reader to Yang et al. (). We provide a novel tool for estimating the likelihood of the samples generated by diffusion SDEs.

#### Typicality vs likelihood.

Theis et al. () observed that likelihood estimates do not correlate with image quality.
Furthermore, deep generative models can assign higher likelihoods to out-of-distribution (OOD) data than the data they were trained on (Choi et al., ; Nalisnick et al., ; Kirichenko et al., ) and therefore perform poorly at OOD detection.
Nalisnick et al. (); Choi et al. () analyze this phenomenon through the lens of typicality, arguing that typical samples do not coincide with the highest likelihood regions.
Ben-Hamu et al. () observed that explicit distortion of an image like inserting a gray patch in the middle may increase the likelihood assigned by a flow-based model. Our investigation contrasts these reports by explicitly studying regions of highest likelihood and shedding light on the probability landscape of diffusion models.

## 8 Conclusion

We provide novel tools for estimating the likelihood for Diffusion SDE samples. Additionally, we theoretically and empirically analyze the estimation error and discuss when exact likelihood estimation for diffusion SDEs might be possible.
These tools, combined with a theoretical mode-seeking analysis, allowed us to study high-probability regions of diffusion models. We made a surprising observation that unnatural and blurry images occupy the highest-density regions of diffusion models.

This work not only enhances the understanding of diffusion model probability landscapes but also opens avenues for improved sample generation strategies.

## Appendix A Notation and assumptions

**Table 1: Summary of notation and abbreviations.**
| Notation | Description |
| --- | --- |
| $D$ | dimensionality of the data |
| $\|{\bm{x}}\|^{2}=\sum_{i}x_{i}^{2}$ | Squared Euclidean norm of ${\bm{x}}\in\mathbb{R}^{D}$ |
| $\nabla_{\bm{x}}$, or $\frac{\partial}{\partial{\bm{x}}}$ | gradient w.r.t ${\bm{x}}$ |
| $\operatorname{div}_{\bm{x}}=\sum_{i}\frac{\partial}{\partial x_{i}}$ | divergence operator w.r.t ${\bm{x}}$ |
| $\mathbb{E}_{{\bm{x}}\sim p}f({\bm{x}})$ | expectation $=\int f({\bm{x}})p({\bm{x}})d{\bm{x}}$ |
| ${\bm{I}}_{D}$ | $D$-dimensional identity matrix |
| $D_{KL}(p||q)$ | Kullback-Leibler divergence $=\mathbb{E}_{{\bm{x}}\sim p}\left(\log p({\bm{x}})-\log q({\bm{x}})\right)$ |
| $f\in\mathcal{C}^{k}$ | function $f$ has continuous derivatives of up to order $k$ |
| $\Delta_{\bm{x}}=\sum_{i}\frac{\partial^{2}}{\partial x_{i}^{2}}$ | Laplace operator |
| $p_{0}$ | data distribution |
| $p_{t}$ | marginal distribution of a process defined by Forward SDE equation 1 |
| $p({\bm{x}}_{t}|{\bm{x}}_{0})=p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$ | Forward transition distribution. For linear drift SDE (equation 1): $p({\bm{x}}_{t}|{\bm{x}}_{0})=\mathcal{N}({\bm{x}}_{t}|\alpha_{t}{\bm{x}}_{0},<br>\sigma_{t}^{2}{\bm{I}}_{D})$ |
| $p({\bm{x}}_{0}|{\bm{x}}_{t})=p_{0|t}({\bm{x}}_{0}|{\bm{x}}_{t})$ | Denoising distribution $p_{0|t}({\bm{x}}_{0}|{\bm{x}}_{t})=p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})p_{0}({<br>\bm{x}}_{0})/p_{t}({\bm{x}}_{t})$ |
| $\mathrm{SNR}(t)=\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}$ | signal to noise ratio |
| $\lambda_{t}=\log\mathrm{SNR}(t)$ | $\log$ signal to noise ratio |
| $\nabla_{\bm{x}}\log p_{t}({\bm{x}})$ | “True” score function |
| ${\bm{s}}(t,{\bm{x}})$ | “approximate” score function |
| $p^{\mathrm{SDE}}_{t}$ | marginal distribution of a process defined by the approximate reverse SDE equation 7 |
| $p^{\mathrm{ODE}}_{t}$ | marginal distribution of a process defined by the approximate ODE equation 6 |
| W | Wiener process running forward in time |
| $\overline{{\textnormal{W}}}$ | Wiener process running backwards in time |

In equation 1, the drift term is linear in ${\bm{x}}$, which corresponds to most commonly used SDEs, because it admits Gaussian transition kernels $p_{t|s}$ for $s<t$ (Song et al., ; ; Kingma et al., ; Kingma & Gao, )
and covers many common implementations of diffusion models (Ho et al., ; Song et al., ; Nichol & Dhariwal, ; Dhariwal & Nichol, ; Karras et al., ).

However, more general SDEs have been proposed that do not assume linear drift (Zhang & Chen, ; Bartosh et al., ):

$$ $d{\bm{x}}_{t}=f(t,{\bm{x}}_{t})dt+g(t)d{\textnormal{W}}_{t}$ (23) $$

with $f$ and $g$ satisfying assumptions below. We define the approximate reverse SDE in the general case

$$ $d{\bm{x}}_{t}=\left(f(t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_{t})\right) dt+g(t)d\overline{{\textnormal{W}}}_{t}.$ (24) $$

In our theorems in section 2 and section 3, we do not assume the linearity of the drift and provide more general formulas. However, results in section 4 only hold for linear drift SDE.
We follow Song et al. () and Lu et al. () and make the following assumptions in our proofs to ensure existence of reverse-time SDEs and correctness of integration by parts.

- 1.
$p_{0}\in\mathcal{C}^{3}$ and $\mathbb{E}_{{\bm{x}}\sim p_{0}}[\|{\bm{x}}\|^{2}]<\infty$.
- 2.
$\forall t\in[0,T]:f(\cdot,t)\in\mathcal{C}^{2}$. And $\exists C>0,\forall{\bm{x}}\in\mathbb{R}^{D},t\in[0,T]:\|f(t,{\bm{x}})\|_{2}
\leq C(1+\|{\bm{x}}\|_{2})$.
- 3.
$\exists C>0,\forall{\bm{x}},{\bm{y}}\in\mathbb{R}^{D}:\|f(t,{\bm{x}})-f(t,{\bm
{y}})\|_{2}\leq C\|{\bm{x}}-{\bm{y}}\|_{2}$.
- 4.
$g\in\mathcal{C}$ and $\forall t\in[0,T],|g(t)|>0$.
- 5.
For any open bounded set $\mathcal{O}$, $\int_{0}^{T}\int_{\mathcal{O}}\left(\|p_{t}({\bm{x}})\|^{2}+D\cdot g(t)^{2}\|
\nabla p_{t}(x)\|^{2}d{\bm{x}}\right)dt<\infty$.
- 6.
$\exists C>0,\forall{\bm{x}}\in\mathbb{R}^{D},t\in[0,T]:\|\nabla p_{t}(x)\|^{2}
\leq C(1+\|{\bm{x}}\|)$.
- 7.
$\exists C>0,\forall{\bm{x}},{\bm{y}}\in\mathbb{R}^{D}:\|\nabla\log p_{t}({\bm{
x}})-\nabla\log p_{t}({\bm{y}})\|\leq C\|{\bm{x}}-{\bm{y}}\|$.
- 8.
$\exists C>0,\forall{\bm{x}}\in\mathbb{R}^{D},t\in[0,T]:\|{\bm{s}}(t,{\bm{x}})
\|\leq C(1+\|{\bm{x}}\|)$.
- 9.
$\exists C>0,\forall{\bm{x}},{\bm{y}}\in\mathbb{R}^{D}:\|{\bm{s}}(t,{\bm{x}})-{
\bm{s}}(t,{\bm{y}})\|\leq C\|{\bm{x}}-{\bm{y}}\|$.
- 10.
Novikov’s condition:
$\mathbb{E}\left[\exp\left(\frac{1}{2}\int_{0}^{T}\|\nabla\log p_{t}({\bm{x}})-
{\bm{s}}(t,{\bm{x}})\|^{2}dt\right)\right]<\infty.$
- 11.
$\forall t\in[0,T],\exists k>0:p_{t}({\bm{x}}),p_{t}^{\mathrm{SDE}}({\bm{x}}),p
_{t}^{\mathrm{ODE}}({\bm{x}})\in O(e^{-\|{\bm{x}}\|^{k}})\text{ as }\|{\bm{x}}
\|\to\infty$.

Additionally, we assume

- 12.
$\forall{\bm{x}}_{0}\in\mathbb{R}^{D}:\mathbb{E}_{{\bm{x}}\sim\nu({\bm{x}}_{0})
}\left(\int_{0}^{T}g^{2}(t)\|\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}
_{t})\|^{2}dt\right)<\infty$, where $\nu({\bm{x}}_{0})$ is the path measure of equation 1 starting at ${\bm{x}}_{0}$.
- 13.
$\mathbb{E}_{{\bm{x}}\sim\nu^{\mathrm{SDE}}}\left(\int_{0}^{T}g^{2}(t)\|\nabla_
{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})-{\bm{s}}(t,{\bm{x}}_{t})\|^{2}
dt\right)<\infty$, where $\nu^{\mathrm{SDE}}$ is the path measure of equation 7.
- 14.
$\forall{\bm{x}}_{0}\in\mathbb{R}^{D},\forall t\in[0,T],\exists k>0:p_{t|0}({
\bm{x}}|{\bm{x}}_{0})\in O(e^{-\|{\bm{x}}\|^{k}})\text{ as }\|{\bm{x}}\|\to\infty$. This trivially holds for the linear drift SDE (equation 1), where $p_{t|0}({\bm{x}}|{\bm{x}}_{0})$ is Gaussian.

## Appendix B Fokker Planck Equation

A useful tool in some of the proofs is the Fokker-Planck equation (Fokker, ; Planck, ; Øksendal & Øksendal, ; Särkkä & Solin, ), a partial differential equation (PDE) governing the evolution of the marginal density of a diffusion process.
For a process described in Equation 23, the evolution of marginal density is described by the Fokker-Planck Equation:

$$ $\frac{\partial}{\partial t}p_{t}({\bm{x}})=-\operatorname{div}\left(f(t,{\bm{x }})p_{t}({\bm{x}})\right)+\frac{1}{2}g^{2}(t)\Delta_{\bm{x}}p_{t}({\bm{x}}),$ (25) $$

where $p_{t}$ is the marginal density at time $t$, which holds for all $t\in(0,T)$ and ${\bm{x}}\in\mathbb{R}^{D}$. Equivalently,

$$ $\begin{split}\frac{\partial}{\partial t}\log p_{t}({\bm{x}})&=-\operatorname{ div}\left(f(t,{\bm{x}})\right)+\frac{1}{2}g^{2}(t)\Delta_{\bm{x}}\log p_{t}({ \bm{x}})\\ &-\nabla_{\bm{x}}\log p_{t}({\bm{x}})^{T}\left(f(t,{\bm{x}})-\frac{1}{2}g^{2}( t)\nabla_{\bm{x}}\log p_{t}({\bm{x}})\right).\end{split}$ (26) $$

## Appendix C Itô’s lemma

The main tool for studying the dynamics of log-density of stochastic processes we will use in our proofs is Itô’s lemma (Itô, ), which states that for a stochastic process

$$ $d{\bm{x}}_{t}=f(t,{\bm{x}}_{t})dt+g(t)d{\textnormal{W}}_{t}$ (27) $$

and a smooth function $h:\mathbb{R}\times\mathbb{R}^{D}\to\mathbb{R}$ it holds that

$$ $dh(t,{\bm{x}}_{t})=\left(\frac{\partial}{\partial t}h(t,{\bm{x}}_{t})+\frac{ \partial}{\partial{\bm{x}}}h(t,{\bm{x}}_{t})^{T}f(t,{\bm{x}}_{t})+\frac{1}{2}g ^{2}(t)\Delta_{\bm{x}}h(t,{\bm{x}}_{t})\right)dt+g(t)\frac{\partial}{\partial{ \bm{x}}}h(t,{\bm{x}}_{t})d{\textnormal{W}}_{t}.$ (28) $$

A more general version of Itô’s lemma holds, which does not assume isotropic diffusion, but we do not need it in our proofs.

## Appendix D Proof of Theorem 2

See 2
For the non-linear drift (equation 23), we have (the difference highlighted in blue)

$$ $F(t,{\bm{x}}_{t})=-\operatorname{div}_{\bm{x}}\left({\color[rgb]{0,0,1} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}f(t,{\bm{x}}_{t})}-g^{2}(t) \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\right)+\frac{1}{2}g^ {2}(t)\|\definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{ rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\|^{2}$ $$

###### Proof.

We will apply Itô’s lemma (Appendix C) to $h(t,{\bm{x}})\coloneqq\log p_{t}({\bm{x}})$.
From Equation 26, we have

$$ $\frac{\partial}{\partial t}h(t,{\bm{x}})=-\mathrm{div}_{\bm{x}}f(t,{\bm{x}})+ \frac{1}{2}g^{2}(t)\Delta_{\bm{x}}\log p_{t}({\bm{x}})-\nabla_{\bm{x}}\log p_{ t}({\bm{x}})^{T}\left(f(t,{\bm{x}})-\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log p_{ t}({\bm{x}})\right)$ (29) $$

and

$$ $\begin{split}&\frac{\partial}{\partial t}h(t,{\bm{x}}_{t})+\frac{\partial}{ \partial{\bm{x}}}h(t,{\bm{x}}_{t})^{T}f(t,{\bm{x}}_{t})+\frac{1}{2}g^{2}(t) \Delta_{\bm{x}}h(t,{\bm{x}}_{t})\\ &=-\mathrm{div}_{\bm{x}}f(t,{\bm{x}})+\frac{1}{2}g^{2}(t)\Delta_{\bm{x}}\log p _{t}({\bm{x}})-\nabla_{\bm{x}}\log p_{t}({\bm{x}})^{T}\left(\cancel{f(t,{\bm{x }})}-\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}})\right)\\ &+\cancel{\nabla_{\bm{x}}\log p_{t}({\bm{x}})^{T}f(t,{\bm{x}}_{t})}+\frac{1}{2 }g^{2}(t)\Delta_{\bm{x}}\log p_{t}({\bm{x}}_{t})\\ &=-\mathrm{div}_{\bm{x}}f(t,{\bm{x}}_{t})+\frac{1}{2}g^{2}(t)\|\nabla_{\bm{x}} \log p_{t}({\bm{x}}_{t})\|^{2}+g^{2}(t)\Delta_{\bm{x}}\log p_{t}({\bm{x}}_{t}) \\ &=-\mathrm{div}_{\bm{x}}\left(f(t,{\bm{x}}_{t})-g^{2}(t)\nabla_{\bm{x}}\log p_ {t}({\bm{x}}_{t})\right)+\frac{1}{2}g^{2}(t)\|\nabla_{\bm{x}}\log p_{t}({\bm{x }}_{t})\|^{2}.\end{split}$ (30) $$

Finally, we get

$$ $\begin{split}d\log p_{t}({\bm{x}}_{t})=&\left(-\mathrm{div}_{\bm{x}}\left(f(t, {\bm{x}}_{t})-g^{2}(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\right)+\frac{1}{ 2}g^{2}(t)\|\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\|^{2}\right)dt\\ &+g(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}d{\textnormal{W}}_{t}.\end{split}$ (31) $$

∎

###### Proof.

## Appendix E Proof of Theorem 1

See 1
For the non-linear drift (equation 23), we have (the difference highlighted in blue)

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \log p_{t}({\bm{x}}_{t})\end{bmatrix}=\begin{bmatrix}{\color[rgb]{0,0,1} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}f(t,{\bm{x}}_{t})}-g^{2}(t) \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\ -{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1} \operatorname{div}_{\bm{x}}f(t,{\bm{x}}_{t})}-\frac{1}{2}g^{2}(t)\| \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\|^{2}\end{bmatrix}dt +g(t)\begin{bmatrix}{\bm{I}}_{D}\\ \definecolor{tcbcolback}{rgb}{1,0.91,0.82}\definecolor{tcbcolframe}{rgb}{1,1,1 }\definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to53.73pt{\vbox to16.85pt{ \pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill {0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ } \nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{} {}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1} \pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}} }{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{} \pgfsys@moveto{0.0pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt} \pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525 pt}\pgfsys@lineto{49.45787pt}{16.84525pt}\pgfsys@curveto{51.81499pt}{16.84525 pt}{53.72575pt}{14.93448pt}{53.72575pt}{12.57736pt}\pgfsys@lineto{53.72575pt}{ 4.26788pt}\pgfsys@curveto{53.72575pt}{1.91077pt}{51.81499pt}{0.0pt}{49.45787pt }{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0 pt}{1.91077pt}{0.0pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{ 1,0.91,0.82}\pgfsys@color@rgb@fill{1}{0.91}{0.82}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{1.4 2262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt}\pgfsys@curveto{1.42262 pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42262pt}\pgfsys@lineto{49 .45787pt}{15.42262pt}\pgfsys@curveto{51.02928pt}{15.42262pt}{52.30313pt}{14.14 877pt}{52.30313pt}{12.57736pt}\pgfsys@lineto{52.30313pt}{4.26788pt} \pgfsys@curveto{52.30313pt}{2.69647pt}{51.02928pt}{1.42262pt}{49.45787pt}{1.42 262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1.42262pt }{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath\pgfsys@fill \pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{ {{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke{ } \pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke{ } \hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})$}}}}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d \overline{{\textnormal{W}}}_{t}.$ $$

###### Proof.

The reverse SDE (Equation 2) can be equivalently written, as

$$ $d{\bm{x}}_{t}=\underbrace{\left(-f(T-t,{\bm{x}}_{t})+g^{2}(T-t)\nabla_{\bm{x}} \log p_{T-t}({\bm{x}}_{t})\right)}_{\eqqcolon\mu(t,{\bm{x}}_{t})}dt+g(T-t)d{ \textnormal{W}}_{t}$ (32) $$

for a forward running brownian motion W and positive $dt$. We will apply Itô’s lemma (Appendix C) to $g(t,{\bm{x}})=\log p_{T-t}({\bm{x}})$.
Since forward and reverse SDEs share marginals we can use Equation 29:

$$ $\begin{split}\frac{\partial}{\partial t}g(t,{\bm{x}})&=\mathrm{div}_{\bm{x}}(f (T-t,{\bm{x}}))-\frac{1}{2}g^{2}(T-t)\Delta_{\bm{x}}\log p_{T-t}({\bm{x}})\\ &+\nabla_{\bm{x}}\log p_{T-t}({\bm{x}})^{T}\left(f(T-t,{\bm{x}})-\frac{1}{2}g^ {2}(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}})\right)\end{split}$ (33) $$

and

$$ $\begin{split}&\frac{\partial}{\partial t}g(t,{\bm{x}}_{t})+\frac{\partial}{ \partial{\bm{x}}}g(t,{\bm{x}}_{t})^{T}\mu(t,{\bm{x}}_{t})+\frac{1}{2}g^{2}(T-t )\Delta_{\bm{x}}g(t,{\bm{x}}_{t})\\ &=\mathrm{div}_{\bm{x}}(f(T-t,{\bm{x}}))-\color[rgb]{1,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{1,0,0}\cancel{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\frac{1}{2}g^{2}(T-t)\Delta_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})}\\ &+\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})^{T}\left(\cancel{f(T-t,{\bm{x}}_{t })}-\frac{1}{2}g^{2}(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})\right)\\ &-\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})^{T}\left(\cancel{f(T-t,{\bm{x}}_{t })}-g^{2}(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})\right)\\ &+\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\cancel{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\frac{1}{2}g^{2}(T-t) \Delta_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})}\\ &=\mathrm{div}_{\bm{x}}(f(T-t,{\bm{x}}_{t}))+\frac{1}{2}g^{2}(T-t)\|\nabla_{ \bm{x}}\log p_{T-t}({\bm{x}}_{t})\|^{2}.\end{split}$ (34) $$

Remarkably, the terms involving the higher order derivatives: $\Delta_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})$ cancel out. Thus, we have

$$ $\begin{split}d\log p_{T-t}({\bm{x}}_{t})=&\left(\mathrm{div}_{\bm{x}}(f(T-t,{ \bm{x}}_{t}))+\frac{1}{2}g^{2}(T-t)\|\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t}) \|^{2}\right)dt\\ &+g(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})^{T}d{\textnormal{W}}_{t}, \end{split}$ (35) $$

which can equivalently be written as

$$ $d\log p_{t}({\bm{x}}_{t})=\left(-\mathrm{div}_{\bm{x}}(f(t,{\bm{x}}_{t}))- \frac{1}{2}g^{2}(t)\|\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\|^{2}\right)dt+g( t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}d\overline{{\textnormal{W}}}_{t},$ (36) $$

where $\overline{{\textnormal{W}}}$ is running backwards in time and $dt$ is negative.
∎

###### Proof.

## Appendix F General SDEs

Karras et al. () showed that there is a more general SDE formulation than Equation 23, which can be interpreted as a continuum between the PF-ODE and SDE formulations.
Specifically, they showed that for any choice of $\beta:\mathbb{R}_{+}\to\mathbb{R}_{+}$ the SDE

$$ $d{\bm{x}}_{t}=\left(f(t,{\bm{x}}_{t})-\left(\frac{1}{2}-\beta(t)\right)g^{2}(t )\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\right)dt+\sqrt{2\beta(t)}g(t)d{ \textnormal{W}}_{t}$ (37) $$

has the same marginals as Equation 23 and it has a reverse-time SDE

$$ $d{\bm{x}}_{t}=\left(f(t,{\bm{x}}_{t})-\left(\frac{1}{2}+\beta(t)\right)g^{2}(t )\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\right)dt+\sqrt{2\beta(t)}g(t)d \overline{{\textnormal{W}}}_{t}.$ (38) $$

One can therefore replace the original model ($\beta\equiv\frac{1}{2}$) with any choice of non-negative $\beta$.
We now derive the augmented dynamics of $\log p_{t}({\bm{x}}_{t})$ for any $\beta$.

### F.1 General Forward Augmented Dynamics

###### Proposition 1 .

For ${\bm{x}}$ following Equation 37 we have

$$ $\begin{split}d\log p_{t}({\bm{x}}_{t})&=\left(-\mathrm{div}_{\bm{x}}f(t,{\bm{x }}_{t})+\left(\frac{1}{2}+\beta(t)\right)g^{2}(t)\Delta_{\bm{x}}\log p_{t}({ \bm{x}}_{t})+\beta(t)g^{2}(t)\|\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\|^{2} \right)dt\\ &+\sqrt{2\beta(t)}g(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}d{\textnormal {W}}_{t}.\end{split}$ (39) $$

Note that since $\beta(t)\geq 0$, the higher order term $\Delta_{\bm{x}}\log p_{t}({\bm{x}}_{t})$ is always non-zero for any $\beta$.

###### Proof.

We proceed in the same way as the proof of Theorem 2 and apply Itô’s lemma (Appendix C) to $h(t,{\bm{x}})=\log p_{t}({\bm{x}})$.
Since Equation 37 shares marginals with Equation 23 we can reuse the derivation of $\frac{\partial}{\partial t}h(t,{\bm{x}})$ from Equation 29:

$$ $\frac{\partial}{\partial t}h(t,{\bm{x}})=-\mathrm{div}_{\bm{x}}f(t,{\bm{x}})+ \frac{1}{2}g^{2}(t)\Delta_{\bm{x}}\log p_{t}({\bm{x}})-\nabla_{\bm{x}}\log p_{ t}({\bm{x}})^{T}\left(f(t,{\bm{x}})-\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log p_{ t}({\bm{x}})\right).$ $$

Therefore for ${\bm{x}}$ following Equation 37:

$$ $\begin{split}d\log p_{t}({\bm{x}}_{t})=&\left(\frac{\partial}{\partial t}h(t,{ \bm{x}}_{t})+\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}\left(f(t,{\bm{x}}_{t} )-\left(\frac{1}{2}-\beta(t)\right)g^{2}(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_ {t})\right)\right.\\ &+\beta(t)g^{2}(t)\Delta\log p_{t}({\bm{x}}_{t})\left.\right)dt+\sqrt{2\beta(t )}g(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}d{\textnormal{W}}_{t}\\ =&\left(-\mathrm{div}_{\bm{x}}f(t,{\bm{x}}_{t})+\left(\frac{1}{2}+\beta(t) \right)g^{2}(t)\Delta_{\bm{x}}\log p_{t}({\bm{x}}_{t})+\beta(t)g^{2}(t)\| \nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\|^{2}\right)dt\\ &+\sqrt{2\beta(t)}g(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}d{\textnormal {W}}_{t}.\end{split}$ (40) $$

∎

###### Proposition 1 .

###### Proof.

### F.2 General Reverse Augmented Dynamics

###### Proposition 2 .

For ${\bm{x}}$ following Equation 38, we have

$$ $\begin{split}d\log p_{t}({\bm{x}}_{t})=&-\left(\mathrm{div}_{\bm{x}}f(t,{\bm{x }}_{t})+\left(\beta(t)-\frac{1}{2}\right)g^{2}(t)\Delta_{\bm{x}}\log p_{t}({ \bm{x}}_{t})+\beta(t)g^{2}(t)\|\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})\|^{2} \right)dt\\ &+\sqrt{2\beta(t)}g(t)\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})^{T}d\overline{{ \textnormal{W}}}_{t}\end{split}$ (41) $$

Note that $\beta\equiv\frac{1}{2}$ (corresponding to Equation 2) is the only choice for which the higher order term involving $\Delta_{\bm{x}}\log p_{t}({\bm{x}}_{t})$ disappears.

###### Proof.

Similarly to the proof of Theorem 1 rewrite the general reverse SDE with positive $dt$ and W going forward in time

$$ $\begin{split}d{\bm{x}}_{t}=&-\left(f(T-t,{\bm{x}}_{t})-\left(\frac{1}{2}+\beta (T-t)\right)g^{2}(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})\right)dt\\ &+\sqrt{2\beta(T-t)}g(T-t)d{\textnormal{W}}_{t}\end{split}$ (42) $$

We apply Itô’s lemma to $g(t,x)=\log p_{T-t}({\bm{x}})$:

$$ $\begin{split}dg(t,{\bm{x}}_{t})=&\frac{\partial}{\partial t}g(t,{\bm{x}}_{t}) dt\\ &-\nabla_{\bm{x}}g(t,{\bm{x}}_{t})^{T}\left(f(T-t,{\bm{x}}_{t})-\left(\frac{1} {2}+\beta(T-t)\right)g^{2}(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})\right )dt\\ &+\beta(T-t)g^{2}(T-t)\Delta_{\bm{x}}g(t,{\bm{x}}_{t})dt\\ &+\sqrt{2\beta(T-t)}g(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})^{T}d{ \textnormal{W}}_{t}\\ =&\left(\mathrm{div}_{\bm{x}}f(T-t,{\bm{x}}_{t})+\left(\beta(T-t)-\frac{1}{2} \right)g^{2}(T-t)\Delta_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})\right)dt\\ &+\beta(T-t)g^{2}(T-t)\|\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})\|^{2}dt\\ &+\sqrt{2\beta(T-t)}g(T-t)\nabla_{\bm{x}}\log p_{T-t}({\bm{x}}_{t})^{T}d{ \textnormal{W}}_{t},\end{split}$ (43) $$

which we rewrite equivalently with $dt<0$ and $\overline{{\textnormal{W}}}$ running backward in time to obtain Equation 41.
∎

###### Proposition 2 .

###### Proof.

## Appendix G Approximate model dynamics

Analogously to Theorem 2 and Theorem 1 we can derive the dynamics of $\log p_{t}^{\mathrm{SDE}}({\bm{x}})$.

###### Theorem 6 (Approximate augmented forward SDE) .

Let ${\bm{x}}$ be a random process defined by Equation 23. Then

$$ $d\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})=G(t,{\bm{x}}_{t})dt+\nabla_{\bm{x}} \log p_{t}^{\mathrm{SDE}}({\bm{x}})^{T}d{\textnormal{W}}_{t},$ (44) $$

where

$$ $G(t,{\bm{x}})=-\mathrm{div}_{\bm{x}}\left(f(t,{\bm{x}})-g^{2}(t){\bm{s}}(t,{ \bm{x}})\right)+\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})\|^{2}-\frac{1}{2}g^{ 2}(t)\|\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})-{\bm{s}}(t,{\bm{x}}) \|^{2}$ (45) $$

###### Proof.

Lu et al. () showed that the corresponding forward SDE to Equation 24 is given by

$$ $d{\bm{x}}_{t}=\left(f(t,{\bm{x}}_{t})+g^{2}(t)\left(\nabla_{\bm{x}}\log p_{t}^ {\mathrm{SDE}}({\bm{x}}_{t})-{\bm{s}}(t,{\bm{x}}_{t})\right)\right)dt+g(t)d{ \textnormal{W}}_{t}.$ (46) $$

We will apply Itô’s lemma (Appendix C) to $h(t,{\bm{x}})=\log p_{t}^{\mathrm{SDE}}({\bm{x}})$ for ${\bm{x}}$ following Equation 23 (not Equation 46, which is intractable due to presence of $\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}$).
$\frac{\partial}{\partial t}h(t,{\bm{x}})$ can be evaluated using Equation 26

$$ $\begin{split}&\frac{\partial}{\partial t}h(t,{\bm{x}})=-\mathrm{div}_{\bm{x}}f (t,{\bm{x}})-g^{2}(t)\left(\Delta_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})- \operatorname{div}_{\bm{x}}{\bm{s}}(t,{\bm{x}})\right)+\frac{1}{2}g^{2}(t) \Delta_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})\\ &-\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})^{T}\left(f(t,{\bm{x}})+g^ {2}(t)\left(\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})-{\bm{s}}(t,{\bm {x}})\right)-\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{ x}})\right)\\ &=-\mathrm{div}_{\bm{x}}f(t,{\bm{x}})-\frac{1}{2}g^{2}(t)\Delta_{\bm{x}}\log p _{t}^{\mathrm{SDE}}({\bm{x}})+g^{2}(t)\operatorname{div}_{\bm{x}}{\bm{s}}(t,{ \bm{x}})\\ &-\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})^{T}\left(f(t,{\bm{x}}_{t} )-g^{2}(t){\bm{s}}(t,{\bm{x}})+\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log p_{t}^{ \mathrm{SDE}}({\bm{x}})\right)\end{split}$ (47) $$

Therefore, we have

$$ $\begin{split}&\frac{\partial}{\partial t}h(t,{\bm{x}})+\nabla_{\bm{x}}h(t,{\bm {x}})f(t,{\bm{x}})+\frac{1}{2}g^{2}(t)\Delta_{\bm{x}}h(t,{\bm{x}})\\ &=-\mathrm{div}_{\bm{x}}f(t,{\bm{x}})+g^{2}(t)\operatorname{div}_{\bm{x}}{\bm{ s}}(t,{\bm{x}})\\ &-\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})^{T}\left(-g^{2}(t){\bm{s} }(t,{\bm{x}})+\frac{1}{2}g^{2}(t)\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm {x}})\right)\\ &=-\mathrm{div}_{\bm{x}}\left(f(t,{\bm{x}})-g^{2}(t){\bm{s}}(t,{\bm{x}})\right )+\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})\|^{2}-\frac{1}{2}g^{2}(t)\|\nabla_ {\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})-{\bm{s}}(t,{\bm{x}})\|^{2}.\end{split}$ (48) $$

Thus for ${\bm{x}}$ following Equation 1, we have

$$ $d\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})=G(t,{\bm{x}}_{t})dt+\nabla_{\bm{x}} \log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})^{T}d{\textnormal{W}}_{t},$ (49) $$

where

$$ $G(t,{\bm{x}})=-\mathrm{div}_{\bm{x}}\left(f(t,{\bm{x}})-g^{2}(t){\bm{s}}(t,{ \bm{x}})\right)+\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})\|^{2}-\frac{1}{2}g^{ 2}(t)\|\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})-{\bm{s}}(t,{\bm{x}}) \|^{2}$ (50) $$

∎

Interestingly, Theorem 6 can be used to derive a lower bound for the likelihood of an individual data point ${\bm{x}}_{0}$ (Kingma et al., ; Song et al., ).

###### Proposition 3 (ELBO for non-linear SDE) .

For any ${\bm{x}}_{0}\in\mathbb{R}^{D}$ and $p_{t}^{\mathrm{SDE}}$ marginal distribution of a process defined by some $p_{T}^{\mathrm{SDE}}$ and equation 24 for $t<T$, we have

$$ $\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})=\underbrace{\frac{T}{2}\mathbb{E}_{t,{ \bm{x}}_{t}}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p^{\mathrm{ SDE}}_{t}({\bm{x}}_{t})\|^{2}}_{\geq 0}+\mathrm{ELBO}({\bm{x}}_{0}),$ (51) $$

where $t\sim\mathcal{U}(0,T)$, ${\bm{x}}_{t}\sim p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$ and

$$ $\mathrm{ELBO}({\bm{x}}_{0})=\mathbb{E}_{{\bm{x}}_{T}\sim p_{T|0}({\bm{x}}_{T}| {\bm{x}}_{0})}[\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T})]+T\mathbb{E}_{t,{\bm{x} }_{t}}L(t,{\bm{x}}_{t})$ (52) $$

and $L(t,{\bm{x}})=-\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})\|^{2}+L_{i}(t,{\bm{x}})$, where one may choose any of the following $L_{1},L_{2},L_{3}$ (one could also have different definitions depending on $t$):

$$ $\displaystyle L_{1}(t,{\bm{x}})$ $\displaystyle=\operatorname{div}_{\bm{x}}\left(f(t,{\bm{x}})-g^{2}(t){\bm{s}}( t,{\bm{x}})\right)$ (53) $\displaystyle L_{2}(t,{\bm{x}})$ $\displaystyle=-\left(f(t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_{t})\right) ^{T}\nabla_{{\bm{x}}_{t}}\log p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$ (54) $\displaystyle L_{3}(t,{\bm{x}})$ $\displaystyle=\operatorname{div}_{\bm{x}}(f(t,{\bm{x}}))+g^{2}(t){\bm{s}}(t,{ \bm{x}}_{t})^{T}\nabla_{{\bm{x}}_{t}}\log p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$ (55) $$

###### Proof.

Using Equation 44:

$$ $\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})=\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T} )-\int_{0}^{T}G(t,{\bm{x}}_{t})dt-\int_{0}^{T}g(t)\nabla_{\bm{x}}\log p^{ \mathrm{SDE}}_{t}({\bm{x}}_{t})^{T}d{\textnormal{W}}_{t},$ (56) $$

for ${\bm{x}}$ being a random trajectory following Equation 23 starting at ${\bm{x}}_{0}$.
Using the definition of $G(t,{\bm{x}})$:

$$ $\begin{split}\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})=&\log p^{\mathrm{SDE}}_{T }({\bm{x}}_{T})\\ &-\int_{0}^{T}\left(-\mathrm{div}_{\bm{x}}\left(f(t,{\bm{x}}_{t})-g^{2}(t){\bm {s}}(t,{\bm{x}}_{t})\right)+\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})\|^{2 }\right)dt\\ &+\frac{1}{2}\int_{0}^{T}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}} \log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})\|^{2}dt\\ &-\int_{0}^{T}g(t)\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})^{T}d{ \textnormal{W}}_{t}.\end{split}$ (57) $$

We can take the expectation of both sides of Equation 57 w.r.t. ${\bm{x}}\sim\nu({\bm{x}}|{\bm{x}}_{0})$, where $\nu({\bm{x}}|{\bm{x}}_{0})$ is a path measure of ${\bm{x}}$ starting at ${\bm{x}}_{0}$.
Note that the LHS of Equation 57 is constant w.r.t $\nu({\bm{x}}|{\bm{x}}_{0})$ and thus it is equal to its expectation.

$$ $\begin{split}\underbrace{\mathbb{E}[\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})]}_ {=\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})}=&\underbrace{\mathbb{E}[\log p^{ \mathrm{SDE}}_{T}({\bm{x}}_{T})]}_{\text{``first term"}}\\ &-\underbrace{\mathbb{E}\left[\int_{0}^{T}\left(-\mathrm{div}_{\bm{x}}\left(f( t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_{t})\right)+\frac{1}{2}g^{2}(t)\|{ \bm{s}}(t,{\bm{x}}_{t})\|^{2}\right)dt\right]}_{\text{``second term"}}\\ &+\underbrace{\mathbb{E}\left[\frac{1}{2}\int_{0}^{T}g^{2}(t)\|{\bm{s}}(t,{\bm {x}}_{t})-\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})\|^{2}dt\right ]}_{\text{``third term"}}\\ &-\underbrace{\mathbb{E}\left[\int_{0}^{T}g(t)\nabla_{\bm{x}}\log p_{t}({\bm{x }}_{t})^{T}d{\textnormal{W}}_{t}\right]}_{\text{``fourth term"}}.\end{split}$ (58) $$

###### Theorem 6 (Approximate augmented forward SDE) .

###### Proof.

###### Proposition 3 (ELBO for non-linear SDE) .

###### Proof.

#### First term.

Since the expectation is taken w.r.t $\nu({\bm{x}}|{\bm{x}}_{0})$, we have

$$ $\mathbb{E}_{{\bm{x}}\sim\nu({\bm{x}}|{\bm{x}}_{0})}[\log p^{\mathrm{SDE}}_{T}( {\bm{x}}_{T})]=\mathbb{E}_{{\bm{x}}_{T}\sim p_{T|0}({\bm{x}}_{T}|{\bm{x}}_{0}) }[\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T})],$ (59) $$

where $p_{T|0}$ is the forward transition probability of equation 23.

#### Second term.

Using Fubini’s theorem we have

$$ $\begin{split}\mathbb{E}_{{\bm{x}}\sim\nu({\bm{x}}|{\bm{x}}_{0})}\left[\int_{0} ^{T}F(t,{\bm{x}}_{t})dt\right]&=\int_{0}^{T}\left(\mathbb{E}_{{\bm{x}}_{t}\sim \nu({\bm{x}}_{t}|{\bm{x}}_{0})}F(t,{\bm{x}}_{t})\right)dt\\ &=\int_{0}^{T}\left(\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({\bm{x}}_{t}|{\bm{x}} _{0})}F(t,{\bm{x}}_{t})\right)dt.\end{split}$ (60) $$

After substituting for $F$, we get

$$ $\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})}F(t,{\bm{x}}_{ t})=\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})}\left(- \mathrm{div}_{\bm{x}}\left(f(t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_{t}) \right)+\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})\|^{2}\right).$ (61) $$

Note that for any $t$ the divergence term under the expectation can equivalently be written in one of three ways (integration by parts; assumptions 8 and 14 in Appendix A):

$$ $\begin{split}&-\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0}) }\operatorname{div}_{\bm{x}}\left(f(t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x} }_{t})\right)\\ &\stackrel{{\scriptstyle(i)}}{{=}}\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({\bm{x} }_{t}|{\bm{x}}_{0})}\left[\left(f(t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_ {t})\right)^{T}\nabla_{{\bm{x}}_{t}}\log p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0}) \right]\\ &\stackrel{{\scriptstyle(ii)}}{{=}}\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({\bm{x }}_{t}|{\bm{x}}_{0})}\left[-\operatorname{div}_{\bm{x}}f(t,{\bm{x}}_{t})-g^{2} (t){\bm{s}}(t,{\bm{x}}_{t})^{T}\nabla_{{\bm{x}}_{t}}\log p_{t|0}({\bm{x}}_{t}| {\bm{x}}_{0})\right],\end{split}$ (62) $$

which holds due to applying integration by parts either

- •
$(i):$ to $f(t,{\bm{x}}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_{t})$ and $p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$, or
- •
$(ii):$ to ${\bm{s}}(t,{\bm{x}}_{t})$ and $p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$.

#### Third term.

$$ $\begin{split}&\mathbb{E}\left[\frac{1}{2}\int_{0}^{T}g^{2}(t)\|{\bm{s}}(t,{\bm {x}}_{t})-\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})\|^{2}dt\right ]\\ &=\frac{1}{2}\int_{0}^{T}g^{2}(t)\left(\mathbb{E}_{{\bm{x}}_{t}\sim p_{t|0}({ \bm{x}}_{t}|{\bm{x}}_{0})}\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p^{ \mathrm{SDE}}_{t}({\bm{x}}_{t})\|^{2}\right)dt\end{split}$ (63) $$

#### Fourth term.

Using Assumption 12 (Appendix A)
and the fact that $g(t)\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})$ is W adapted, we have

$$ $\mathbb{E}\left[\int_{0}^{T}g(t)\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{ x}}_{t})^{T}d{\textnormal{W}}_{t}\right]=0.$ (64) $$

Combining all four terms yields the claim.
∎

###### Corollary 1 (ELBO for Linear SDE) .

For an SDE with linear drift (equation 1), for any ${\bm{x}}_{0}\in\mathbb{R}^{D}$, assuming $p^{\mathrm{SDE}}_{T}=\mathcal{N}(\mathbf{0},\sigma_{T}^{2}{\bm{I}}_{D})$ we have

$$ $\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})=\underbrace{\frac{T}{2}\mathbb{E}_{t,{ \bm{\varepsilon}}}g^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p^{ \mathrm{SDE}}_{t}({\bm{x}}_{t})\|^{2}}_{\geq 0}+\mathrm{ELBO}({\bm{x}}_{0})$ (65) $$

where $t\sim\mathcal{U}(0,T)$, ${\bm{\varepsilon}}\sim\mathcal{N}(\mathbf{0},{\bm{I}}_{D})$, ${\bm{x}}_{t}=\alpha_{t}{\bm{x}}_{0}+\sigma_{t}{\bm{\varepsilon}}$ and

$$ $\mathrm{ELBO}({\bm{x}}_{0})=C-\frac{e^{\lambda_{\text{min}}}}{2}\|{\bm{x}}_{0} \|^{2}-\frac{T}{2}\mathbb{E}_{t,{\bm{\varepsilon}}}\left(-\frac{d\lambda_{t}}{ dt}\right)\|\sigma_{t}{\bm{s}}(t,\alpha_{t}{\bm{x}}_{0}+\sigma_{t}{\bm{ \varepsilon}})+{\bm{\varepsilon}}\|^{2}$ (66) $$

and $C=-\frac{D}{2}\left(1+\log(2\pi\sigma_{0}^{2})\right)$.

###### Proof.

In the linear SDE case (equation 1) we have $p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})=\mathcal{N}({\bm{x}}_{t}|\alpha_{t}{\bm{x}}
_{0},\sigma_{t}^{2}{\bm{I}}_{D})$. Using 3, we have

$$ $\mathrm{ELBO}({\bm{x}}_{0})=\mathbb{E}_{{\bm{x}}_{T}\sim p_{T|0}({\bm{x}}_{T}| {\bm{x}}_{0})}[\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T})]+T\mathbb{E}_{t,{\bm{x} }_{t}}L(t,{\bm{x}}_{t})$ (67) $$

and we choose

$$ $\begin{split}L(t,{\bm{x}})=&-\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})\|^{2}+L _{3}(t,{\bm{x}})\\ =&-\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})\|^{2}+\operatorname{div}_{\bm{x}} (f(t,{\bm{x}}))+g^{2}(t){\bm{s}}(t,{\bm{x}})^{T}\nabla_{{\bm{x}}}\log p_{t|0}( {\bm{x}}|{\bm{x}}_{0})\\ =&f(t)D-\frac{1}{2}g^{2}(t)\|{\bm{s}}(t,{\bm{x}})-\nabla_{{\bm{x}}}\log p_{t|0 }({\bm{x}}|{\bm{x}}_{0})\|^{2}+\frac{1}{2}g^{2}(t)\|\nabla_{{\bm{x}}}\log p_{t |0}({\bm{x}}|{\bm{x}}_{0})\|^{2}\end{split}$ (68) $$

Since $\nabla_{{\bm{x}}}\log p_{t|0}({\bm{x}}|{\bm{x}}_{0})=\frac{\alpha_{t}{\bm{x}}_
{0}-{\bm{x}}}{\sigma_{t}^{2}}$ and ${\bm{x}}_{t}\sim p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})$ is equivalent to ${\bm{x}}_{t}=\alpha_{t}{\bm{x}}_{0}+\sigma_{t}{\bm{\varepsilon}}$ for ${\bm{\varepsilon}}\sim\mathcal{N}(\mathbf{0},{\bm{I}}_{D})$, we have $\nabla_{{\bm{x}}}\log p_{t|0}({\bm{x}}_{t}|{\bm{x}}_{0})=\frac{-{\bm{
\varepsilon}}}{\sigma_{t}}$ and

$$ $\begin{split}\mathrm{ELBO}({\bm{x}}_{0})=&-\frac{T}{2}\mathbb{E}_{t,{\bm{ \varepsilon}}}g^{2}(t)\left\|{\bm{s}}(t,\alpha_{t}{\bm{x}}_{0}+\sigma_{t}{\bm{ \varepsilon}})+\frac{{\bm{\varepsilon}}}{\sigma_{t}}\right\|^{2}\\ &+\mathbb{E}_{{\bm{x}}_{T}\sim p_{T|0}({\bm{x}}_{T}|{\bm{x}}_{0})}[\log p^{ \mathrm{SDE}}_{T}({\bm{x}}_{T})]+DT\mathbb{E}_{t,{\bm{x}}_{t}}f(t)+\frac{T}{2} \mathbb{E}_{t,{\bm{\varepsilon}}}g^{2}(t)\left\|\frac{{\bm{\varepsilon}}}{ \sigma_{t}}\right\|^{2}\\ =&-\frac{T}{2}\mathbb{E}_{t,{\bm{\varepsilon}}}\left(-\frac{d\lambda_{t}}{dt} \right)\left\|\sigma_{t}{\bm{s}}(t,\alpha_{t}{\bm{x}}_{0}+\sigma_{t}{\bm{ \varepsilon}})+{\bm{\varepsilon}}\right\|^{2}\\ &+\underbrace{\mathbb{E}_{{\bm{\varepsilon}}}[\log p^{\mathrm{SDE}}_{T}(\alpha _{T}{\bm{x}}_{0}+\sigma_{T}{\bm{\varepsilon}})]}_{\text{``first term"}}+ \underbrace{DT\mathbb{E}_{t}\left(f(t)-\frac{1}{2}\frac{d\lambda_{t}}{dt} \right)}_{\text{``second term"}}\end{split}$ (69) $$

###### Corollary 1 (ELBO for Linear SDE) .

###### Proof.

#### First term

$$ $\begin{split}\mathbb{E}_{{\bm{\varepsilon}}}[\log p^{\mathrm{SDE}}_{T}(\alpha_ {T}{\bm{x}}_{0}+\sigma_{T}{\bm{\varepsilon}})]=&-\frac{D}{2}\log(2\pi\sigma_{T }^{2})-\frac{1}{2\sigma_{T}^{2}}\mathbb{E}_{{\bm{\varepsilon}}}\|\alpha_{T}{ \bm{x}}_{0}+\sigma_{T}{\bm{\varepsilon}}\|^{2}\\ =&-\frac{D}{2}\log(2\pi\sigma_{T}^{2})-\frac{1}{2\sigma_{T}^{2}}\left(\alpha_{ T}^{2}\|{\bm{x}}_{0}\|^{2}+\sigma_{T}^{2}D\right)\\ =&-\frac{D}{2}\left(1+\log(2\pi\sigma_{T}^{2})\right)-\frac{e^{\lambda_{ \mathrm{min}}}}{2}\|{\bm{x}}_{0}\|^{2}\end{split}$ (70) $$

#### Second term

$$ $DT\mathbb{E}_{t}\left(f(t)-\frac{1}{2}\frac{d\lambda_{t}}{dt}\right)=D\int_{0} ^{T}\frac{d\log\sigma_{t}}{dt}dt=D\left(\log\sigma_{T}-\log\sigma_{0}\right)$ (71) $$

Combining all the terms yields the claim.
∎
We can now use Theorem 6 to prove Theorem 4.
See 4
Furthermore, ${\textnormal{Y}}_{{\bm{x}}_{0}}$ can be written as ${\textnormal{Y}}_{{\bm{x}}_{0}}={\textnormal{Y}}_{1}+{\textnormal{Y}}_{2}$, where

$$ ${\textnormal{Y}}_{1}=\frac{1}{2}\int_{0}^{T}g^{2}(t)\|\nabla_{\bm{x}}\log p_{t }^{\mathrm{SDE}}({\bm{x}}_{t})-{\bm{s}}(t,{\bm{x}}_{t})\|^{2}dt$ $$

and

$$ $\mathbb{E}{\textnormal{Y}}_{2}=0;\mathrm{Var}({\textnormal{Y}}_{2})=\int_{0}^{ T}g^{2}(t)\mathbb{E}_{{\bm{x}}_{t}\sim p({\bm{x}}_{t}|{\bm{x}}_{0})}\|{\bm{s}} (t,{\bm{x}}_{t})-\nabla_{\bm{s}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^{2}dt.$ $$

For the non-linear drift (equation 23), we have (the difference highlighted in blue)

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \omega_{t}\end{bmatrix}=\begin{bmatrix}{\color[rgb]{0,0,1}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}f(t,{\bm{x}}_{t})}\\ -{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1} \operatorname{div}_{\bm{x}}f(t,{\bm{x}}_{t})}+g^{2}(t)\Big{(}\frac{1}{2}\| \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to31.41pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{27.14684pt}{16 .84525pt}\pgfsys@curveto{29.50395pt}{16.84525pt}{31.41472pt}{14.93448pt}{31.41 472pt}{12.57736pt}\pgfsys@lineto{31.41472pt}{4.26788pt}\pgfsys@curveto{31.4147 2pt}{1.91077pt}{29.50395pt}{0.0pt}{27.14684pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{27.14684pt}{15.42262pt}\pgfsys@curveto{28.71825pt}{15.422 62pt}{29.9921pt}{14.14877pt}{29.9921pt}{12.57736pt}\pgfsys@lineto{29.9921pt}{4 .26788pt}\pgfsys@curveto{29.9921pt}{2.69647pt}{28.71825pt}{1.42262pt}{27.14684 pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1 .42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\|^{2}+\definecolor{tcbcolback}{rgb}{ 1,0.85,0.85}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{ rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode \hbox to23.66pt{\vbox to15pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0 .0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{10.72736pt}\pgfsys@curveto{0.0pt}{13.08447pt}{1.91077pt} {14.99524pt}{4.26788pt}{14.99524pt}\pgfsys@lineto{19.38907pt}{14.99524pt} \pgfsys@curveto{21.74619pt}{14.99524pt}{23.65695pt}{13.08447pt}{23.65695pt}{10 .72736pt}\pgfsys@lineto{23.65695pt}{4.26788pt}\pgfsys@curveto{23.65695pt}{1.91 077pt}{21.74619pt}{0.0pt}{19.38907pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{1,0.85,0.85}\pgfsys@color@rgb@fill{1}{0 .85}{0.85}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}} }{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{} {{}}}{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{10.7273 6pt}\pgfsys@curveto{1.42262pt}{12.29877pt}{2.69647pt}{13.57262pt}{4.26788pt}{1 3.57262pt}\pgfsys@lineto{19.38907pt}{13.57262pt}\pgfsys@curveto{20.96048pt}{13 .57262pt}{22.23433pt}{12.29877pt}{22.23433pt}{10.72736pt}\pgfsys@lineto{22.234 33pt}{4.26788pt}\pgfsys@curveto{22.23433pt}{2.69647pt}{20.96048pt}{1.42262pt}{ 19.38907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69 647pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{4.62817pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces$\operatorname{div}_{\bm{x}}$}}}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{} \hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss }}\lxSVG@closescope\endpgfpicture}}\definecolor{tcbcolback}{rgb}{0.9,0.9,1} \definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{tcbcol@origin}{rgb}{0,0,0} \definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\leavevmode\hbox to31.41 pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to 0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ } \pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt} \pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{}\pgfsys@beginscope \pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{pgffillcolor}{rgb}{1,1,1} \pgfsys@color@gray@fill{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0pt}{4.26788pt} \pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448pt}{1.91077pt} {16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{27.14684pt}{16.84525pt} \pgfsys@curveto{29.50395pt}{16.84525pt}{31.41472pt}{14.93448pt}{31.41472pt}{12 .57736pt}\pgfsys@lineto{31.41472pt}{4.26788pt}\pgfsys@curveto{31.41472pt}{1.91 077pt}{29.50395pt}{0.0pt}{27.14684pt}{0.0pt}\pgfsys@lineto{4.26788pt}{0.0pt} \pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{27.14684pt}{15.42262pt}\pgfsys@curveto{28.71825pt}{15.422 62pt}{29.9921pt}{14.14877pt}{29.9921pt}{12.57736pt}\pgfsys@lineto{29.9921pt}{4 .26788pt}\pgfsys@curveto{29.9921pt}{2.69647pt}{28.71825pt}{1.42262pt}{27.14684 pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647pt}{1 .42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\Big{)}\end{bmatrix}dt+g(t)\begin{bmatrix}{ \bm{I}}_{D}\\ \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d{\textnormal{W}}_{t}.$ $$

###### Proof.

Using Theorem 6 we have

$$ $\begin{split}\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})&=\log p^{\mathrm{SDE}}_{T }({\bm{x}}_{T})-\int_{0}^{T}G(t,{\bm{x}}_{t})dt-\int_{0}^{T}g(t)\nabla_{\bm{x} }\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})^{T}d{\textnormal{W}}_{t},\\ &=\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T})-\int_{0}^{T}d\omega_{t}+{\textnormal {Y}}_{1}+{\textnormal{Y}}_{2},\end{split}$ (72) $$

where

$$ ${\textnormal{Y}}_{1}=\frac{1}{2}\int_{0}^{T}g^{2}(t)\|\nabla_{\bm{x}}\log p_{t }^{\mathrm{SDE}}({\bm{x}}_{t})-{\bm{s}}(t,{\bm{x}}_{t})\|^{2}dt$ (73) $$

and

$$ ${\textnormal{Y}}_{2}=\int_{0}^{T}g(t)\left({\bm{s}}(t,{\bm{x}}_{t})-\nabla_{ \bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})\right)d{\textnormal{W}}_{t}.$ (74) $$

Since $\int_{0}^{T}d\omega_{t}=\omega_{T}-\omega_{0}=\omega_{T}$, we have

$$ $\log p^{\mathrm{SDE}}_{0}({\bm{x}}_{0})=\log p^{\mathrm{SDE}}_{T}({\bm{x}}_{T} )-\omega_{T}+{\textnormal{Y}}$ (75) $$

for ${\textnormal{Y}}={\textnormal{Y}}_{1}+{\textnormal{Y}}_{2}$.
∎

Similarly to Theorem 6 we can derive the dynamics of $\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$ under the approximate reverse SDE (Equation 24).

###### Theorem 7 (Approximate augmented reverse SDE) .

Let ${\bm{x}}$ be a random process following Equation 24, then

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ \log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\end{bmatrix}=\begin{bmatrix}f(t,{\bm{x }}_{t})-g^{2}(t){\bm{s}}(t,{\bm{x}}_{t})\\ \tilde{F}(t,{\bm{x}}_{t})\end{bmatrix}dt+g(t)\begin{bmatrix}{\bm{I}}_{d}\\ \nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})^{T}\end{bmatrix}d \overline{{\textnormal{W}}}_{t},$ (76) $$

where

$$ $\tilde{F}(t,{\bm{x}})=-\operatorname{div}_{\bm{x}}f(t,{\bm{x}})-g^{2}(t) \underbrace{\left(\Delta_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})- \operatorname{div}_{\bm{x}}{\bm{s}}(t,{\bm{x}})\right)}_{=0\text{ when }{\bm{s }}(t,x)=\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})}-\frac{1}{2}g^{2}(t )\|\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})\|^{2}.$ (77) $$

###### Proof.

The approximate reverse SDE can equivalently be written as

$$ $d{\bm{x}}_{t}=-\left(f(T-t,{\bm{x}}_{t})-g^{2}(T-t){\bm{s}}(T-t,{\bm{x}}_{t}) \right)dt+g(T-t)d{\textnormal{W}}_{t}$ (78) $$

for $dt>0$ and W running forward in time. We will apply Itô’s lemma (Appendix C) to $h(t,{\bm{x}})=\log p_{T-t}^{\mathrm{SDE}}({\bm{x}})$.
From Equation 47 we have:

$$ $\begin{split}&\frac{\partial}{\partial t}h(t,{\bm{x}})=\operatorname{div}_{\bm {x}}f(T-t,{\bm{x}})+\frac{1}{2}g^{2}(T-t)\Delta_{\bm{x}}\log p_{T-t}^{\mathrm{ SDE}}({\bm{x}})-g^{2}(T-t)\operatorname{div}_{\bm{x}}{\bm{s}}(T-t,{\bm{x}})\\ &+\nabla_{\bm{x}}\log p_{T-t}^{\mathrm{SDE}}({\bm{x}})^{T}\left(f(T-t,{\bm{x}} )-g^{2}(T-t){\bm{s}}(T-t,{\bm{x}})+\frac{1}{2}g^{2}(T-t)\nabla_{\bm{x}}\log p_ {T-t}^{\mathrm{SDE}}({\bm{x}})\right)\end{split}$ (79) $$

Therefore the drift of $h(t,{\bm{x}}_{t})$ is given by

$$ $\begin{split}&\frac{\partial}{\partial t}h(t,{\bm{x}}_{t})+\nabla_{\bm{x}}h(t, {\bm{x}}_{t})^{T}\left(-f(T-t,{\bm{x}}_{t})+g^{2}(T-t){\bm{s}}(T-t,{\bm{x}}_{t })\right)+\frac{1}{2}g^{2}(T-t)\Delta_{\bm{x}}h(t,{\bm{x}}_{t})\\ &=\operatorname{div}_{\bm{x}}f(T-t,{\bm{x}}_{t})+g^{2}(T-t)\Delta_{\bm{x}}\log p _{T-t}^{\mathrm{SDE}}({\bm{x}}_{t})-g^{2}(T-t)\operatorname{div}_{\bm{x}}{\bm{ s}}(T-t,{\bm{x}}_{t})\\ &+\frac{1}{2}g^{2}(T-t)\|\nabla_{\bm{x}}\log p_{T-t}^{\mathrm{SDE}}({\bm{x}}_{ t})\|^{2}\end{split}$ (80) $$

and therefore for ${\bm{x}}$ following the approximate reverse SDE (Equation 24), we have

$$ $d\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})=\tilde{F}(t,{\bm{x}}_{t})dt+g(t) \nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})^{T}d\overline{{ \textnormal{W}}}_{t},$ (81) $$

where $dt<0$, $\overline{{\textnormal{W}}}$ is running backwards in time and

$$ $\tilde{F}(t,{\bm{x}})=-\operatorname{div}_{\bm{x}}f(t,{\bm{x}})-g^{2}(t)\Delta _{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}})+g^{2}(t)\operatorname{div}_{\bm{x }}{\bm{s}}(t,{\bm{x}})-\frac{1}{2}g^{2}(t)\|\nabla_{\bm{x}}\log p_{t}^{\mathrm {SDE}}({\bm{x}})\|^{2}.$ (82) $$

∎

Theorem 7 defines the exact dynamics of $\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})$.
However, $d\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})$ depends on $\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})$, which we cannot access in practice.
We only have access to the approximation ${\bm{s}}(t,{\bm{x}})$.
We now show that replacing the true $\nabla_{\bm{x}}\log p^{\mathrm{SDE}}_{t}({\bm{x}}_{t})$ with ${\bm{s}}$ no longer provides exact likelihood estimates, but an “upper bound in expectation”.
See 3
Furthermore, X can be written as ${\textnormal{X}}={\textnormal{X}}_{1}+{\textnormal{X}}_{2}$, where

$$ ${\textnormal{X}}_{1}=\int_{0}^{T}g^{2}(t)\left(\operatorname{div}_{\bm{x}}{\bm {s}}(t,{\bm{x}}_{t})+\frac{1}{2}\|{\bm{s}}(t,{\bm{x}}_{t})\|^{2}-\Delta_{\bm{x }}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})-\frac{1}{2}\|\nabla_{\bm{x}}\log p_{ t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^{2}\right)dt$ $$

and

$$ $\mathbb{E}{\textnormal{X}}_{2}=0;\mathrm{Var}({\textnormal{X}}_{2})=\int_{0}^{ T}g^{2}(t)\mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})}\|{ \bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t}) \|^{2}dt$ $$

For the non-linear drift (equation 23), we have (the difference highlighted in blue)

$$ $d\begin{bmatrix}{\bm{x}}_{t}\\ r_{t}\end{bmatrix}=\begin{bmatrix}{\color[rgb]{0,0,1}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,1}f(t,{\bm{x}}_{t})}-g^{2}(t)\definecolor{tcbcolback} {rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1}\definecolor{ tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor{.}{rgb}{ 0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture\makeatletter\hbox {\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ } \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{ 0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{{}{}{}{} \pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[named]{ pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\\ -{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1} \operatorname{div}_{\bm{x}}f(t,{\bm{x}}_{t})}-\frac{1}{2}g^{2}(t)\| \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}\|^{2}\end{bmatrix}dt+g(t)\begin{bmatrix}{\bm {I}}_{D}\\ \definecolor{tcbcolback}{rgb}{0.9,0.9,1}\definecolor{tcbcolframe}{rgb}{1,1,1} \definecolor{tcbcol@origin}{rgb}{0,0,0}\definecolor{.}{rgb}{0,0,0}\definecolor {.}{rgb}{0,0,0}\leavevmode\hbox to33.44pt{\vbox to16.85pt{\pgfpicture \makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope \pgfsys@invoke{ }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0} {0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to 0.0pt{{}{}{}{}\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{}\definecolor[ named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{ } \pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{ }{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}\pgfsys@moveto{0.0 pt}{4.26788pt}\pgfsys@lineto{0.0pt}{12.57736pt}\pgfsys@curveto{0.0pt}{14.93448 pt}{1.91077pt}{16.84525pt}{4.26788pt}{16.84525pt}\pgfsys@lineto{29.16907pt}{16 .84525pt}\pgfsys@curveto{31.52618pt}{16.84525pt}{33.43695pt}{14.93448pt}{33.43 695pt}{12.57736pt}\pgfsys@lineto{33.43695pt}{4.26788pt}\pgfsys@curveto{33.4369 5pt}{1.91077pt}{31.52618pt}{0.0pt}{29.16907pt}{0.0pt}\pgfsys@lineto{4.26788pt} {0.0pt}\pgfsys@curveto{1.91077pt}{0.0pt}{0.0pt}{1.91077pt}{0.0pt}{4.26788pt} \pgfsys@closepath\pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }{}{}{}{}{}{}{}{} \definecolor[named]{pgffillcolor}{rgb}{0.9,0.9,1}\pgfsys@color@rgb@fill{0.9}{0 .9}{1}\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0}\pgfsys@invoke{ }{{}{}{{}}}{{} {}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}}}{}{}{{}{}{{}}}{{}{}{{}} }{}{}\pgfsys@moveto{1.42262pt}{4.26788pt}\pgfsys@lineto{1.42262pt}{12.57736pt} \pgfsys@curveto{1.42262pt}{14.14877pt}{2.69647pt}{15.42262pt}{4.26788pt}{15.42 262pt}\pgfsys@lineto{29.16907pt}{15.42262pt}\pgfsys@curveto{30.74048pt}{15.422 62pt}{32.01433pt}{14.14877pt}{32.01433pt}{12.57736pt}\pgfsys@lineto{32.01433pt }{4.26788pt}\pgfsys@curveto{32.01433pt}{2.69647pt}{30.74048pt}{1.42262pt}{29.1 6907pt}{1.42262pt}\pgfsys@lineto{4.26788pt}{1.42262pt}\pgfsys@curveto{2.69647 pt}{1.42262pt}{1.42262pt}{2.69647pt}{1.42262pt}{4.26788pt}\pgfsys@closepath \pgfsys@fill\pgfsys@invoke{ }\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@fill@opacity{1.0} \pgfsys@invoke{ }{{{}}{{}}{{}}{{}}{{}}{{}}{{}}\pgfsys@beginscope\pgfsys@invoke { }\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{3.42262pt}{5.92262pt}\pgfsys@invoke { }\hbox{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hbox{\set@color{ \ignorespaces${\bm{s}}(t,{\bm{x}}_{t})$}}}}\pgfsys@invoke{\lxSVG@closescope } \pgfsys@endscope}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}^{T}\end{bmatrix}d\overline{{\textnormal{W}}} _{t},$ $$

###### Proof.

$$ $\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})=\log p_{T}^{\mathrm{SDE}}({\bm{x}}_{T} )-\int_{0}^{T}d\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})$ (83) $$

From Theorem 7 we have

$$ $\begin{split}\int_{0}^{T}d\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})=&\int_{0}^{T }\tilde{F}(t,{\bm{x}}_{t})dt+\int_{0}^{T}g(t)\nabla_{\bm{x}}\log p_{t}^{ \mathrm{SDE}}({\bm{x}}_{t})^{T}d\overline{{\textnormal{W}}}_{t}\\ =&\int_{0}^{T}\left(-\operatorname{div}_{\bm{x}}f(t,{\bm{x}}_{t})-\frac{1}{2}g ^{2}(t)\|{\bm{s}}(t,{\bm{x}}_{t})\|^{2}\right)dt\\ &+\int_{0}^{T}g(t){\bm{s}}(t,{\bm{x}}_{t})^{T}d\overline{{\textnormal{W}}}_{t} +{\textnormal{X}}_{1}+{\textnormal{X}}_{2},\end{split}$ (84) $$

where

$$ ${\textnormal{X}}_{1}=\int_{0}^{T}g^{2}(t)\left(\operatorname{div}_{\bm{x}}{\bm {s}}(t,{\bm{x}}_{t})+\frac{1}{2}\|{\bm{s}}(t,{\bm{x}}_{t})\|^{2}-\Delta_{\bm{x }}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})-\frac{1}{2}\|\nabla_{\bm{x}}\log p_{ t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^{2}\right)dt$ (85) $$

and

$$ ${\textnormal{X}}_{2}=\int_{0}^{T}g(t)\left(\nabla_{\bm{x}}\log p_{t}^{\mathrm{ SDE}}({\bm{x}}_{t})-{\bm{s}}(t,{\bm{x}}_{t})\right)^{T}d\overline{{\textnormal {W}}}_{t}$ (86) $$

Note that from Assumption 13 (Appendix A) we have

$$ $\mathbb{E}{\textnormal{X}}_{2}=0.$ (87) $$

Furthermore, using Fubini’s theorem, we have

$$ $\begin{split}\mathbb{E}{\textnormal{X}}_{1}=&\int_{0}^{T}g^{2}(t)\mathbb{E}_{{ \bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})}\left(\operatorname{div}_{ \bm{x}}{\bm{s}}(t,{\bm{x}}_{t})+\frac{1}{2}\|{\bm{s}}(t,{\bm{x}}_{t})\|^{2}- \Delta_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\right.\\ &\left.-\frac{1}{2}\|\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^ {2}\right)dt.\end{split}$ (88) $$

Now rewrite

$$ $\begin{split}&\mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})} \left(\operatorname{div}_{\bm{x}}{\bm{s}}(t,{\bm{x}}_{t})-\Delta_{\bm{x}}\log p _{t}^{\mathrm{SDE}}({\bm{x}}_{t})\right)\\ &=\mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})} \operatorname{div}_{\bm{x}}\left({\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p _{t}^{\mathrm{SDE}}({\bm{x}}_{t})\right)\\ &\stackrel{{\scriptstyle(i)}}{{=}}\mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{ SDE}}({\bm{x}}_{t})}\left(-{\bm{s}}(t,{\bm{x}}_{t})+\nabla_{\bm{x}}\log p_{t}^ {\mathrm{SDE}}({\bm{x}}_{t})\right)^{T}\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE} }({\bm{x}}_{t})\\ &=\mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})}\left(-{\bm{ s}}(t,{\bm{x}}_{t})^{T}\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})+ \|\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^{2}\right),\end{split}$ (89) $$

where $(i)$ is applying integration by parts (Assumptions 8 and 11 in Appendix A). Substituting back to Equation 88, we get

$$ $\begin{split}\mathbb{E}{\textnormal{X}}_{1}&=\int_{0}^{T}\frac{1}{2}g^{2}(t) \mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})}\left(\|{\bm{s }}(t,{\bm{x}}_{t})\|^{2}-2{\bm{s}}(t,{\bm{x}}_{t})^{T}\nabla_{\bm{x}}\log p_{t }^{\mathrm{SDE}}({\bm{x}}_{t})+\|\nabla_{\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm {x}}_{t})\|^{2}\right)dt\\ &=\int_{0}^{T}\frac{1}{2}g^{2}(t)\mathbb{E}_{{\bm{x}}_{t}\sim p_{t}^{\mathrm{ SDE}}({\bm{x}}_{t})}\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_{\bm{x}}\log p_{t}^{ \mathrm{SDE}}({\bm{x}}_{t})\|^{2}dt.\end{split}$ (90) $$

Therefore

$$ $\begin{split}\log p_{0}^{\mathrm{SDE}}({\bm{x}}_{0})=&\log p_{T}^{\mathrm{SDE} }({\bm{x}}_{T})-\int_{0}^{T}\left(-\operatorname{div}_{\bm{x}}f(t,{\bm{x}}_{t} )-\frac{1}{2}g^{2}(t){\bm{s}}(t,{\bm{x}}_{t})\right)dt\\ &-\int_{0}^{T}g(t){\bm{s}}(t,{\bm{x}}_{t})^{T}d\overline{{\textnormal{W}}}_{t} -{\textnormal{X}},\end{split}$ (91) $$

where

$$ $\mathbb{E}{\textnormal{X}}=\int_{0}^{T}\frac{1}{2}g^{2}(t)\mathbb{E}_{{\bm{x}} _{t}\sim p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})}\|{\bm{s}}(t,{\bm{x}}_{t})-\nabla_ {\bm{x}}\log p_{t}^{\mathrm{SDE}}({\bm{x}}_{t})\|^{2}dt\geq 0.$ (92) $$

∎

###### Proof.

###### Theorem 7 (Approximate augmented reverse SDE) .

###### Proof.

###### Proof.

## Appendix H Proof of Theorem 5

In the following sections, we assume the linear drift SDE Equation 1. In Theorem 5 we explicitly assume Gaussian forward transition densities, which are only guaranteed in the linear drift SDE.
See 5
Note that without assuming invertibility of ${\bm{A}}$, Equation 16 becomes

$$ ${\bm{A}}(s,{\bm{y}}_{s})\left(\dot{{\bm{y}}}_{s}-f(s){\bm{y}}_{s}+g^{2}(s) \nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\right)=-\frac{1}{2}g^{2}(s)\nabla_{{ \bm{y}}}\Delta_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})$ (93) $$

###### Proof.

We begin by noting that for linear SDE (equation 1) $p_{t|s}$ is Gaussian for $s<t$ and therefore

$$ $\begin{split}\log p_{s|t}({\bm{y}}_{s}|{\bm{x}}_{t})&=\log p_{t|s}({\bm{x}}_{t }|{\bm{y}}_{s})+\log p_{s}({\bm{y}}_{s})-\log p_{t}({\bm{x}}_{t})\\ &=C-\frac{\|{\bm{x}}_{t}-\tilde{f}(s){\bm{y}}_{s}\|^{2}}{2\tilde{g}^{2}(s)}+ \log p_{s}({\bm{y}}_{s})-\log p_{t}({\bm{x}}_{t}),\end{split}$ (94) $$

where $\tilde{f}(s)=\frac{\alpha_{t}}{\alpha_{s}}$ and $\tilde{g}^{2}(s)=\sigma_{t}^{2}-\tilde{f}^{2}(s)\sigma_{s}^{2}$ (See Appendix A.1 in Kingma et al. ()). Since $p_{s|t}({\bm{y}}_{s}|{\bm{x}}_{t})=\max_{{\bm{x}}_{s}}p_{s|t}({\bm{x}}_{s}|{
\bm{x}}_{t})$, it must hold that

$$ $\nabla_{{\bm{y}}_{s}}\log p_{s|t}({\bm{y}}_{s}|{\bm{x}}_{t})=0\hskip 5.0pt \text{for all}\hskip 5.0pts<t.$ (95) $$

Therefore

$$ $\frac{d}{ds}\left(\nabla_{{\bm{y}}_{s}}\log p_{s|t}({\bm{y}}_{s}|{\bm{x}}_{t}) \right)=0\hskip 5.0pt\text{for all}\hskip 5.0pts<t.$ (96) $$

From Equation 94 we have

$$ $\nabla_{{\bm{y}}_{s}}\log p_{s|t}({\bm{y}}_{s}|{\bm{x}}_{t})=\nabla_{{\bm{y}}_ {s}}\log p_{s}({\bm{y}}_{s})+\frac{\tilde{f}(s)}{\tilde{g}^{2}(s)}\left({\bm{x }}_{t}-\tilde{f}(s){\bm{y}}_{s}\right)=0$ (97) $$

and thus

$$ $\frac{d}{ds}\left(\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s})-\psi(s){\bm{y} }_{s}+\phi(s){\bm{x}}_{t}\right)=0,$ (98) $$

where $\psi(s)=\frac{\tilde{f}^{2}(s)}{\tilde{g}^{2}(s)}$ and $\phi(s)=\frac{\tilde{f}(s)}{\tilde{g}^{2}(s)}$. Note that

$$ $\frac{d}{ds}\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s})=\frac{\partial}{ \partial s}\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s})+\nabla^{2}_{{\bm{y}}_ {s}}\log p_{s}({\bm{y}}_{s})\dot{{\bm{y}}_{s}}$ (99) $$

and we can use Equation 26 to re-write the first term

$$ $\begin{split}&\frac{\partial}{\partial s}\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{ y}}_{s})=\nabla_{{\bm{y}}_{s}}\frac{\partial}{\partial s}\log p_{s}({\bm{y}}_{ s})\\ &=\nabla_{{\bm{y}}_{s}}\left(-f(s)D+\frac{1}{2}g^{2}(s)\Delta_{{\bm{y}}_{s}} \log p_{s}({\bm{y}}_{s})-\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s})^{T}(f(s ){\bm{y}}_{s}-\frac{1}{2}g^{2}(s)\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s}) \right)\\ &=\frac{1}{2}g^{2}(s)\nabla_{{\bm{y}}_{s}}\Delta_{{\bm{y}}_{s}}\log p_{s}({\bm {y}}_{s})-f(s)\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}}_{s}){\bm{y}}_{s}\\ &-f(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})+g^{2}(s)\nabla^{2}_{{\bm{y}}} \log p_{s}({\bm{y}}_{s})\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\\ &=\frac{1}{2}g^{2}(s)\nabla_{{\bm{y}}_{s}}\Delta_{{\bm{y}}_{s}}\log p_{s}({\bm {y}}_{s})-f(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\\ &-\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\left(f(s){\bm{y}}_{s}-g^{2}(s) \nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\right)\end{split}$ (100) $$

and thus

$$ $\begin{split}\frac{d}{ds}\nabla_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s})&=\frac{ 1}{2}g^{2}(s)\nabla_{{\bm{y}}_{s}}\Delta_{{\bm{y}}_{s}}\log p_{s}({\bm{y}}_{s} )-f(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\\ &+\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\left(\dot{{\bm{y}}_{s}}-f(s){ \bm{y}}_{s}+g^{2}(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\right)\end{split}$ (101) $$

For the remaining terms we first note

$$ $\tilde{f}(s)=\frac{\alpha_{t}}{\alpha_{s}}=\exp\{\log\alpha_{t}-\log\alpha_{s} \}=\exp\{\int_{s}^{t}\frac{d}{du}\log\alpha_{u}\}=\exp\{\int_{s}^{t}f(u)\}$ (102) $$

and in particular $\frac{d}{ds}\log\tilde{f}(s)=-f(s)$. Similarly

$$ $\begin{split}\tilde{g}^{2}(s)&=\sigma_{t}^{2}-\tilde{f}^{2}(s)\sigma_{s}^{2}= \alpha_{t}^{2}\left(\frac{\sigma_{t}^{2}}{\alpha_{t}^{2}}-\frac{\sigma_{s}^{2} }{\alpha_{s}^{2}}\right)=\alpha_{t}^{2}\left(e^{-\lambda_{t}}-e^{-\lambda_{s}} \right)=\alpha_{t}^{2}\int_{s}^{t}\frac{d}{du}e^{-\lambda_{u}}du\\ &=\alpha_{t}^{2}\int_{s}^{t}\left(-\frac{d\lambda_{u}}{du}\right)e^{-\lambda_{ u}}du=\alpha_{t}^{2}\int_{s}^{t}\left(-\frac{d\lambda_{u}}{du}\right)\frac{ \sigma_{u}^{2}}{\alpha_{u}^{2}}du=\alpha_{t}^{2}\int_{s}^{t}\frac{g^{2}(u)}{ \alpha_{u}^{2}}du\\ &=\int_{s}^{t}\tilde{f}^{2}(u)g^{2}(u)du\end{split}$ (103) $$

and in particular $\frac{d}{ds}\log\tilde{g}^{2}(s)=\frac{1}{\tilde{g}^{2}(s)}\frac{d}{ds}\tilde{
g}^{2}(s)=-\psi(s)g^{2}(s)$. Therefore

$$ $\begin{split}\frac{d}{ds}\left(-\psi(s){\bm{y}}_{s}+\phi(s){\bm{x}}_{t}\right) &=-\psi^{\prime}(s){\bm{y}}_{s}-\psi(s)\dot{{\bm{y}}_{s}}+\phi^{\prime}(s){\bm {x}}_{t}\\ &=-\psi(s)\dot{{\bm{y}}_{s}}+\phi^{\prime}(s){\bm{x}}_{t}-\left(\phi^{\prime}( s)\tilde{f}(s)-f(s)\psi(s)\right){\bm{y}}_{s}\\ &=-\psi(s)\left(\dot{{\bm{y}}_{s}}-f(s){\bm{y}}_{s}\right)+\phi^{\prime}(s) \left({\bm{x}}_{t}-\tilde{f}(s){\bm{y}}_{s}\right)\\ &=-\psi(s)\left(\dot{{\bm{y}}_{s}}-f(s){\bm{y}}_{s}\right)+\phi(s)\frac{d}{ds} \left(\log\phi(s)\right)\left({\bm{x}}_{t}-\tilde{f}(s){\bm{y}}_{s}\right). \end{split}$ (104) $$

From Equation 97, we have

$$ $\phi(s)\left({\bm{x}}_{t}-\tilde{f}(s){\bm{y}}_{s}\right)=-\nabla_{{\bm{y}}} \log p_{s}({\bm{y}}_{t})$ (105) $$

and

$$ $\frac{d}{ds}\left(\log\phi(s)\right)=\frac{d}{ds}\log\tilde{f}(s)-\frac{d}{ds} \log\tilde{g}^{2}(s)=-f(s)+\psi(s)g^{2}(s).$ (106) $$

Thus

$$ $\begin{split}\frac{d}{ds}\left(-\psi(s){\bm{y}}_{s}+\phi(s){\bm{x}}_{t}\right) &=-\psi(s)\left(\dot{{\bm{y}}}_{s}-f(s){\bm{y}}_{s}\right)+\left(f(s)-\psi(s)g ^{2}(s)\right)\nabla_{\bm{y}}\log p_{s}({\bm{y}}_{s})\\ &=-\psi(s)\left(\dot{{\bm{y}}}_{s}-f(s){\bm{y}}_{s}+g^{2}(s)\nabla_{\bm{y}} \log p_{s}({\bm{y}}_{s})\right)+f(s)\nabla_{\bm{y}}\log p_{s}({\bm{y}}_{s}). \end{split}$ (107) $$

Putting it all together, we have

$$ $\begin{split}0&=\frac{d}{ds}\left(\nabla_{{\bm{y}}}\log p({\bm{y}}_{s}|{\bm{x} }_{t})\right)\\ &=\frac{1}{2}g^{2}(s)\nabla_{{\bm{y}}}\Delta_{{\bm{y}}}\log p_{s}({\bm{y}}_{s} )-f(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\\ &+\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\left(\dot{{\bm{y}}}_{s}-f(s){ \bm{y}}_{s}+g^{2}(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\right)\\ &-\psi(s)\left(\dot{{\bm{y}}}_{s}-f(s){\bm{y}}_{s}+g^{2}(s)\nabla_{\bm{y}}\log p _{s}({\bm{y}}_{s})\right)+f(s)\nabla_{\bm{y}}\log p_{s}({\bm{y}}_{s})\\ &=\frac{1}{2}g^{2}(s)\nabla_{{\bm{y}}}\Delta_{{\bm{y}}}\log p_{s}({\bm{y}}_{s} )+\left(\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})-\psi(s){\bm{I}}_{D} \right)\left(\dot{{\bm{y}}}_{s}-f(s){\bm{y}}_{s}+g^{2}(s)\nabla_{{\bm{y}}}\log p _{s}({\bm{y}}_{s})\right),\end{split}$ (108) $$

or equivalently for ${\bm{A}}(s,{\bm{y}})=\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}})-\psi(s){\bm{I}}
_{D}$

$$ $\dot{{\bm{y}}}_{s}=f(s){\bm{y}}_{s}-g^{2}(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y }}_{s})-\frac{1}{2}g^{2}(s){\bm{A}}(s,{\bm{y}}_{s})^{-1}\nabla_{{\bm{y}}} \Delta_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})$ (109) $$

if ${\bm{A}}(s,{\bm{y}}_{s})$ is invertible for all $s<t$ and

$$ $\begin{split}\psi(s)&=\frac{\tilde{f}^{2}(s)}{\tilde{g}^{2}(s)}=\frac{\alpha_{ t}^{2}}{\alpha_{s}^{2}\left(\sigma_{t}^{2}-\tilde{f}^{2}(s)\sigma_{s}^{2} \right)}=\frac{\alpha_{t}^{2}}{\alpha_{s}^{2}\sigma_{t}^{2}-\alpha_{t}^{2} \sigma_{s}^{2}}=\frac{1}{\alpha_{s}^{2}}\frac{1}{\frac{\sigma_{t}^{2}}{\alpha_ {t}^{2}}-\frac{\sigma_{s}^{2}}{\alpha_{s}^{2}}}\\ &=\frac{1}{\alpha_{s}^{2}}\frac{1}{e^{-\lambda_{t}}-e^{-\lambda_{s}}}=\frac{e^ {\lambda_{s}}}{\alpha_{s}^{2}}\frac{e^{\lambda_{t}}}{e^{\lambda_{s}}-e^{ \lambda_{t}}}=\frac{1}{\sigma_{s}^{2}}\frac{e^{\lambda_{t}}}{e^{\lambda_{s}}-e ^{\lambda_{t}}}.\end{split}$ (110) $$

∎

###### Proof.

## Appendix I Mode-seeking ODE in the Gaussian case

We will prove the claims from 1. We recall it for completeness.
See 1

###### Proof.

We first note that when $p_{0}$ Gaussian and the SDE is linear (equation 1) then $p_{s}$ are Gaussian $\forall s$. In particular $\nabla_{\bm{x}}\Delta_{\bm{x}}\log p_{s}({\bm{x}})=0$ for al $s\in[0,T]$ and ${\bm{x}}\in\mathbb{R}^{D}$. Therefore equation 16 becomes equation 18. We will now study ${\bm{y}}_{s}$ following equation 18. Recalling Equation 101:

$$ $\begin{split}\frac{d}{ds}\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})&=\cancelto{ 0}{\frac{1}{2}g^{2}(s)\nabla_{{\bm{y}}}\Delta_{{\bm{y}}}\log p_{s}({\bm{y}}_{s })}-f(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\\ &+\nabla^{2}_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\left(\dot{{\bm{y}}_{s}}-f(s){ \bm{y}}_{s}+g^{2}(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\right)\\ &=-f(s)\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})+\nabla^{2}_{{\bm{y}}}\log p_{ s}({\bm{y}}_{s})\left(\dot{{\bm{y}}_{s}}-f(s){\bm{y}}_{s}+g^{2}(s)\nabla_{{\bm {y}}}\log p_{s}({\bm{y}}_{s})\right).\end{split}$ (111) $$

If we then assume Equation 18, we have

$$ $\frac{d}{ds}\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})=-f(s)\nabla_{{\bm{y}}} \log p_{s}({\bm{y}}_{s})$ (112) $$

and to simplify further, using the fact that $f(s)=\frac{d}{ds}\log\alpha_{s}$

$$ $\frac{d}{ds}\left(\alpha_{s}\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})\right)= \frac{d}{ds}\alpha_{s}\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})+\alpha_{s} \frac{d}{ds}\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})=0.$ (113) $$

Hence, for ${\bm{y}}_{s}$ satisfying Equation 18, we have

$$ $\alpha_{s}\nabla_{{\bm{y}}}\log p_{s}({\bm{y}}_{s})=\alpha_{t}\nabla_{{\bm{y}} }\log p_{t}({\bm{y}}_{t})\hskip 5.0pt\text{for all}\hskip 5.0pts<t.$ (114) $$

We can thus rewrite

$$ $\begin{split}\dot{{\bm{y}}}_{s}&=f(s){\bm{y}}_{s}-g^{2}(s)\nabla_{{\bm{y}}} \log p_{s}({\bm{y}}_{s})\\ &=f(s){\bm{y}}_{s}-\frac{g^{2}(s)}{\alpha_{s}}\alpha_{s}\nabla_{{\bm{y}}}\log p _{s}({\bm{y}}_{s})\\ &=f(s){\bm{y}}_{s}-\frac{g^{2}(s)}{\alpha_{s}}\alpha_{t}\nabla_{{\bm{y}}}\log p _{t}({\bm{y}}_{t}).\end{split}$ (115) $$

Furthermore

$$ $\begin{split}\frac{d}{ds}\left(\frac{{\bm{y}}_{s}}{\alpha_{s}}\right)=&\frac{ \dot{{\bm{y}}}_{s}\alpha_{s}-{\bm{y}}_{s}\alpha_{s}^{\prime}}{\alpha_{s}^{2}}= \frac{\left(\cancel{\frac{\alpha_{s}^{\prime}}{\alpha_{s}}{\bm{y}}_{s}}-\frac{ g^{2}(s)}{\alpha_{s}}\alpha_{t}\nabla_{{\bm{y}}}\log p_{t}({\bm{y}}_{t})\right )\alpha_{s}-\cancel{{\bm{y}}_{s}\alpha_{s}^{\prime}}}{\alpha_{s}^{2}}\\ &=-\frac{g^{2}(s)}{\alpha_{s}^{2}}\alpha_{t}\nabla_{{\bm{y}}}\log p_{t}({\bm{y }}_{t})=\frac{d\lambda_{s}}{ds}e^{-\lambda_{s}}\alpha_{t}\nabla_{{\bm{y}}}\log p _{t}({\bm{y}}_{t})\end{split}$ (116) $$

and we can solve

$$ $\begin{split}\frac{{\bm{y}}_{t}}{\alpha_{t}}-\frac{{\bm{y}}_{0}}{\alpha_{0}}&= \int_{0}^{t}\left(\frac{d\lambda_{s}}{ds}e^{-\lambda_{s}}\alpha_{t}\nabla_{{ \bm{y}}}\log p_{t}({\bm{y}}_{t})\right)ds=\left(\int_{0}^{t}\frac{d\lambda_{s} }{ds}e^{-\lambda_{s}}ds\right)\alpha_{t}\nabla_{{\bm{y}}}\log p_{t}({\bm{y}}_{ t})\\ &=\left(\int_{\lambda_{\text{max}}}^{\lambda_{t}}e^{-\lambda}d\lambda\right) \alpha_{t}\nabla_{{\bm{y}}}\log p_{t}({\bm{y}}_{t})=\left(e^{-\lambda_{\text{ max}}}-e^{-\lambda_{t}}\right)\alpha_{t}\nabla_{{\bm{y}}}\log p_{t}({\bm{y}}_{ t}).\end{split}$ (117) $$

Leveraging that $\alpha_{0}=1$, we get

$$ $\begin{split}{\bm{y}}_{0}=&\frac{{\bm{y}}_{t}}{\alpha_{t}}+e^{-\lambda_{t}} \alpha_{t}\log p_{t}({\bm{y}}_{t})-e^{-\lambda_{\text{max}}}\alpha_{t}\nabla_{ {\bm{y}}}\log p_{t}({\bm{y}}_{t})\\ =&\frac{{\bm{y}}_{t}+\sigma_{t}^{2}\nabla_{\bm{x}}\log p_{t}({\bm{x}}_{t})}{ \alpha_{t}}-e^{-\lambda_{\text{max}}}\alpha_{t}\nabla_{{\bm{x}}}\log p_{t}({ \bm{x}}_{t})\\ =&\mathbb{E}\left[{\bm{x}}_{0}|{\bm{x}}_{t}\right]-e^{-\lambda_{\text{max}}} \alpha_{t}\nabla_{{\bm{x}}}\log p_{t}({\bm{x}}_{t})\\ =&\operatorname*{arg\,max}_{{\bm{x}}_{0}}p({\bm{x}}_{0}|{\bm{x}}_{t})+\mathcal {O}(e^{-\lambda_{\text{max}}}).\end{split}$ (118) $$

∎

###### Proof.

## Appendix J Non-smooth mode-tracking curve

Figure: Figure 11: The mode tracking curve need not be continuous. Left: mode-tracking curve with a discontinuous jump at $s^{*}$. Right: distribution of $\log p_{s|t}({\bm{x}}_{s}|{\bm{x}}_{t})$ for values around $s^{*}$. At $s=s^{*}$ the $\operatorname*{arg\,max}$ changes discontinuously.
Refer to caption: extracted/5973435/figures/discontinuous-mode-tracking-curve-updated.png

An important assumption in Theorem 5 is that for a fixed $t\in(0,T]$ and a noisy point ${\bm{x}}_{t}\in\mathbb{R}^{D}$, there exists a smooth curve $s\mapsto{\bm{y}}_{s}$ such that

$$ $p_{s|t}({\bm{y}}_{s}|{\bm{x}}_{t})=\max_{{\bm{x}}_{s}}p_{s|t}({\bm{x}}_{s}|{ \bm{x}}_{t}).$ (119) $$

It is an assumption that need not hold.
To demonstrate we define the data distribution as 1D mixture of 3 gaussians $p=\sum_{i=1}^{3}w_{i}\mathcal{N}(\mu_{i},\sigma^{2})$, where $\mu_{1}=-2.5$, $\mu_{2}=-1.5$, $\mu_{3}=1$ and $\sigma^{2}=0.1$ and weights $w_{1}=w_{2}=0.274$ and $w_{3}=0.45$.
We model the distrbution with a VP-SDE (Song et al., ), where $\sigma_{t}^{2}=\frac{1}{1+e^{\lambda_{t}}}=1-\alpha_{t}^{2}$.
We then choose ${\bm{x}}_{t}=-2.5$ and $t$ such that $\lambda_{t}=-8$ and visualize $\log p({\bm{x}}_{s}|{\bm{x}}_{t})$ for all $s<t$ and all ${\bm{s}}_{t}\in[-4,3.5]$ and the mode-tracking curve ${\bm{y}}_{s}|{\bm{x}}_{t}$ in white (Figure 11 left).

The mode-tracking curve exhibits a discontinuous jump at $s^{*}$ such that $\lambda_{s^{*}}\approx 1.28$.
The distribution $p_{s|t}({\bm{x}}_{s}|{\bm{x}}_{t})$ has the mode at ${\bm{x}}_{s}\approx 0.86$ for $s=s^{*}-ds$ and at ${\bm{x}}_{s}\approx-1.81$ for $s=s^{*}+ds$ (Figure 11 right).

## Appendix K Proof of 1

###### Proof.

From Equation 26, we have that

$$ $\frac{\partial}{\partial t}\log p_{t}({\bm{z}})=-\operatorname{div}_{\bm{z}}f_ {1}(t,{\bm{z}})-\nabla_{\bm{z}}\log p_{t}({\bm{z}})^{T}f_{1}(t,{\bm{z}}).$ $$

Therefore

$$ $\displaystyle\frac{d}{dt}\log p_{t}({\bm{z}}_{t})$ $\displaystyle=\frac{\partial}{\partial t}\log p_{t}({\bm{z}}_{t})+\nabla_{\bm{ x}}\log p_{t}({\bm{z}}_{t})^{T}\frac{d}{dt}{\bm{z}}_{t}$ $\displaystyle=-\operatorname{div}_{\bm{z}}f_{1}(t,{\bm{z}}_{t})+\nabla_{\bm{x} }\log p_{t}({\bm{z}}_{t})^{T}\left(f_{2}(t,{\bm{z}}_{t})-f_{1}(t,{\bm{z}}_{t}) \right).$ $$

∎

###### Proof.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - The related work does not take into consideration a very relevant paper "Null-text Guidance in Diffusion Models is
Secretly a Cartoon-style Creator", Zhao et al which is not discussed. In particular it would be relevant to highlight some connection and insights with this previous related work. 

- See weaknesses
soundness: 3
strengths: - The paper is well written and it develops new theoretical insights, especially the augmented SDE for tracking likelihood evolution are innovative and pratical since they don't introduce any additional computational cost.

- The proposed high-probability sampler is a nice tool to generate high-likelihood samples that are not discoverable by traditional sampling techniques

- The paper analize the diffusion probability landscape finding analyzing the different images in high-likelihood regions, experimentally demonstrating that the proposed theoretical tools can be used to perform analyisis of this landscape.
summary: This paper introduces new augmented SDE for likelihood estimation in SDE sampling of diffusion models. Moreover, a novel theoretical mode-tracking approach is proposed in order to locate the exact mode of the generative distribution and introduce a high-probability sampler capable of generating samples with higher likelihood than all the other samples, under theoretically proved guarantees. 
Finally, the main findings of the proposes analysis lies in the discovery of Cartoon-like images that lies in high-likelihood regions, even if the model has never been trained with images of the same style.
weaknesses: - The proposed mode-tracking approach has a very high computational cost and this is not clearly discussed. The discussion on how this mode-tracking approach scale and its computational limitations should be discussed and quantified

- The high-likelihood samples discovered by the proposed analysis does not seem to have a real practical advantage, being cartoon drawings or blurry images. It is not discussed how these insight can be leveraged to improve sample generation strategies or how these high-density samples can be practically useful.

- The limitations of the work are not discussed at all, a limitations section covering all the main potential limitations should be added.

- The work does not report any implementation details, code and implementation are not submitted in the supplementary materials. It would be important to openly release the contribution upon acceptance to ensure reproducibility and improve transparency.

- The analysis of high-density regions is done on small and simple diffusion models trained on restricted datasets with limited variability. Moreover the selected models and study is done on uncoditional sampling without any guidance from text. In my opinion it would be valuable to explore real-world and more complex diffusion models (such as SD, Flux etc), especially focusing on the impact of text.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Does the high-likelihood sampling technique impact the diversity of generated samples, potentially leading to mode collapse or other reductions in output variability? 

2. For applications that prioritize both high likelihood and visual quality, what adjustments to the high-probability sampler could mitigate the production of low-quality images?
soundness: 2
strengths: The paper introduces a novel framework for estimating likelihoods within diffusion models using augmented stochastic differential equations (SDEs) and high-probability samplers. This is an important advancement because it allows the exploration of high-likelihood regions without increasing computational costs. By deriving density estimates through augmented SDEs, the authors provide a theoretically efficient approach to analyzing model outputs across noise levels, setting a foundation for future studies in likelihood-based generative modeling. The findings reveal that high-likelihood samples often resemble cartoonish or blurry images—even though such images are not present in the training data.
summary: This paper investigates the high-density regions of diffusion models, discovering that samples in these regions often appear as unrealistic cartoon-like or blurry images, despite the absence of such images in the training data. The authors propose a novel framework based on augmented stochastic differential equations (SDEs) to estimate the likelihoods of generated samples. This approach enables efficient, high-likelihood sampling without additional computational cost. The authors also introduce a high-probability sampling method that consistently yields higher-likelihood images than traditional sampling techniques, while their empirical analysis shows that images with less detail (e.g., blurred images) tend to achieve higher likelihood scores. The paper contributes to a better understanding of the diffusion model probability landscape and the relationship between likelihood and image quality.
weaknesses: 1. Although the paper sheds light on high-density regions and likelihood estimation, it lacks a clear discussion on how these findings could practically inform the design or improvement of diffusion models in applied settings. Without recommendations on balancing high-likelihood sampling with image quality, it’s challenging to draw valuable insights from the findings, particularly for practitioners focused on real-world applications.

2. The paper describes the emergence of cartoon-like images in high-likelihood samples but provides limited exploration into why or how this phenomenon occurs. 

3. The experiments are primarily conducted on well-known datasets like CIFAR-10 and FFHQ-256. Expanding the study to other types of diffusion models or additional data domains would strengthen the generalizability of the findings. Additionally, clarifying any specific architectural limitations of the proposed high-probability sampler could benefit those aiming to extend the approach.

4. While the paper emphasizes the likelihood of generated samples, it does not address diversity within these high-likelihood outputs. The authors mention blurry and cartoon-like images, but they do not provide an assessment of diversity or how the high-likelihood sampling might affect the overall variance in generated outputs. It’s unclear if the method leads to mode collapse or reduces the richness of the model’s output space.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Have the authors thought about how the proposed theory can be used to improve the quality of image generation?
2. Can we use the proposed estimation for the log probability to evaluate model quality the same way as [1] (Table 2)? Will there be any difference between SDE and ODE estimation? Which one is better to compare different models?

[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020c.
soundness: 4
strengths: * The authors develop a novel theory of augmented SDEs. They provided a clear and detailed derivation and coupled it with the bias estimation 
* Landscape analysis gives valuable insight into the structure of high-probability samples and provides a theoretical justification for the known fact that distorted images tend to have a higher likelihood 
* The paper is well-structured and easy to read
summary: The authors offer a theoretical framework for estimating the log probability of samples that follow SDE in diffusion models. They develop novel forward and reverse augmented dynamics that estimates log probability not only for the case where $\nabla \log p_{t}(x)$ is known precisely but for the case where the score function is known. Then, the authors provide novel upper and lower bounds on the bias of the proposed estimator and analyze them on the experimental data. Lastly, the authors apply their proposed theory to analyze the diffusion probability landscape. They propose a simple yet effective way to generate samples with the highest log probability and show that such samples are unrealistic and blurry.
weaknesses: 1. It isn't clear whether analysis from Section 5 can lead to the creation of better stochastic samplers or improve the quality of image generation. Overall practical implications of this work are quite poor 
2. There is no intuition behind observations from Figure 4. More precisely, what does it mean that the model optimized for sample quality yields a smaller difference between $p_{0}^{ODE}$ and $p_{0}^{SDE}$?
3. Experiments were conducted on rather small and outdated diffusion models. It would greatly improve the scope of the work if the experiments were performed on the frontier models

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 4
questions: Please see weaknesses
soundness: 3
strengths: - The paper is well written and the use of good figures and colors in equations really helps the presentation
- They present a stochastic way to keep track of the likelihood and a likelihood maximizing ODE
summary: The paper presents a perspective on studying likelihoods for diffusion models. They introduce a stochastic way to keep track of the likelihood. They introduce a likelihood maximizing ODE that maximizes the likelihood of a path for any given starting condition. They attempt to argue that they can use this ODE to demonstrate that the high likelihood region of diffusion models results in cartoon like images.
weaknesses: - Up to section 4.2 everything was very reasonable and nicely introduced, however the use of the ODE in remark 1 for sampling is a big mistake. Real world data is multimodal, for instance CIFAR 10 has at least 10 modes. To use the assumption that the data is gaussian in order to use remark 1 results in incorrect results. Specifically this ODE doesn't sample from the correct target distribution, and therefore we observe those cartoon like images. 
- Additionally when comparing the likelihood of images sampled normally or from the HP-ODE the likelihoods were measured using different methods. In section 4.3 it is explained how they compute the likelihood of $p(y_0|x_t)$. Empirically they show that it results in higher likelihood than $p(x_0 |x_t)$, however since these likelihoods are being evaluated in different ways this comparison doesn't make sense. If done correctly the sample $y_0$ would be evaluated using the same way as $x_0$, so we would evaluate the likelihood of the **same distribution**
- The paper has a limited amount of quantitative evidence
- There is no explanation on the hyperparameters that were used during sampling

Meta-review: This paper proposes a mode-tracking method to sample data from high-density regions from a diffusion model. To this end, a likelihood estimation method is proposed and theoretical guarantees are given. Empirically, this paper found that the high likelihood examples are more cartoon-like drawings or blurry images. The reviewers agree that the paper has novel technical and theoretical contributions, but the application of the proposed method is unclear. The authors have partially addressed this concern in the rebuttal. Overall, I think it is a solid paper and recommend accepting it to ICLR.

Decision: Accept (Poster)