Diffusion Models as Cartoonists: The Curious Case of High Density Regions
Authors: Rafal Karczewski
Markus Heinonen
Vikas Garg
Venue: ICLR 2025 Poster (2025)
Keywords: diffusion models
likelihood estimation
Primary area: generative models
TL;DR: We develop novel methods for likelihood estimation for stochastic sampling in diffusion models. We use them to study high probability regions.

Abstract: We investigate what kind of images lie in the high-density regions of diffusion models. We introduce a theoretical mode-tracking process capable of pinpointing the exact mode of the denoising distribution, and we propose a practical high-density sampler that consistently generates images of higher likelihood than usual samplers. Our empirical findings reveal the existence of significantly higher likelihood samples that typical samplers do not produce, often manifesting as cartoon-like drawings or blurry images depending on the noise level. Curiously, these patterns emerge in datasets devoid of such examples. We also present a novel approach to track sample likelihoods in diffusion SDEs, which remarkably incurs no additional computational cost. Code is available at https://github.com/Aalto-QuML/high-density-diffusion

Published as a conference paper at ICLR 2025
DIFFUSION MODELS AS CARTOONISTS: THE CURIOUS CASE OF HIGH DENSITY REGIONS
Rafal Karczewski, Markus Heinonen
Aalto University
(rafal karczewski, markus heinonen Baa
8aa tto.f.... fi
Vikas Garg
Yai Ltd and Aalto University
vgargecsail mim.meet
ABSTRACT
We investigate what kind of images lie in the high-density regions of diffu- sion models. We introduce theoretical mode-tracking process capable of pin- pointing the exact mode of the denoising distribution, and we propose a prac- tical high-density sampler that consistently generates images of higher likeli- hood than usual samplers. Our empirical findings reveal the existence of sig- nificantly higher likelihood samples that typical samplers do not produce, often manifesting as cartoon-like drawings blurry images depending on the noise level. Curiously, these patterns emerge in datasets devoid of such examples. We also present a novel approach to track sample likelihoods in diffusion SDEs, which remarkably incurs no additional computational cost. Code is available /ects::/ggitie com Aalto CLL0-QMMLL hi gh density -i-ttyy-iii diffiioon
INTRODUCTION
Recently, Karras et al. (2024a) attributed the empirical success of guided diffusion models to their ability to limit outliers, i.c. samples Vo ~ Po with low likelihood po We argue that these models in fact also elude samples with very high likelihoods. Our assertion stems from investigating key question: what manifests if we bias the sampler towards high-density regions of po? However, an immediate hurdle is the inability of stochastic diffusion models to track their own likelihood (Song ct al. b)).
First, we show that likelihood be tracked in diffusion models with novel augmented stochastic differential equations (SDE), which govern the evolution of a sample with its log-density log pt (ar)
0.95
0.90
0.60
= 0.55
== 0.50
0:45
0.10
T= 0:30
= 0.20
A= 0.10
~ p(N
E(xu|xx)
High density No p(suie) posubo)
Figure 1: High-density samples Vo|w, resemble cartoon drawings. This is in contrast to regular denoising samples æp p( ax.) or expectations E[xy Ja.]]. The data contains cartoons.

---

Published as a conference paper at ICLR 2025
Summary of our contributions
Augmented SDEs (section section
Exact log Pe (xx) when V log Pr(2) and pr are known Estimated log pi (1;,) for approximate V log (ar)
subsection 2.2
subsection 2.3
Variational gap log Po SDB
ELBO(xz) formula
Equation 13
Estimation error analysis
subsection 3.2
Mode-tracking (section 4):
Evolution of the mode of (1))
Theorem 5
High-Density ODE approximation
Remark
General instantaneous change of variables
Lemma
Diffusion probability landscape (section 5)
Unrealistic images have the highest likelihoods Blurring an image increases its likelihood Likelihood estimates correlate with PNG size
Figure
Figure 10
Figure 8
Figure 2: Our contributions.
under the optimal (unknown) model. For approximate models, we provide a formula for the bias in the log-densit, estimate. The evaluation of the log-density estimate comes no additional cost, and can be used with any pretrained model without further tuning.
Then, introduce theoretical mode-tracking process, which finds the exact mode of the denoising distribution p(zo|a;; (z2) under some technical assumptions, albeit at high computational cost. We propose an approximative high density sampler whose images almost always have higher likelihood than regular samples.
We leverage these findings to give insights into the diffusion model probability landscape. We find that dramatically higher likelihood samples exist that regular sampler never returns in practise. We observe that the high density samples tend to be (i) blank images for high noise levels, (ii) cartoon drawings for moderate noise, and (iii) blurry images for low noise (See Figure Dp. This is despite the datasets not containing any cartoon drawings. Curiously, on FFHQ-256 we observe 97% correlation between model's likelihood estimates and the amount of information in the image.
We summarize our contributions
Figure
2 LIKELIHOOD ESTIMATION IN SDE MODELS
We model the data distribution with a diffusion process (Song et al. 2021c). which gradually trans- forms the data into Gaussian. We define a forward process p(æ, xc) N(x;;; a,æo, Ip)..
where D is the dimensionality of the data, a₁ and a, are schedule parameters with a decreasing signal-to-noise ratio SNR(t) at fo exv Song al. (2021c) showed an equivalence linear stochastic differential equation (SDE) with Zp ~ data distribution and
dx; = f(t)x + g(t)dW
(1)
of, and (W,) a Wiener process. Most of our theoretical results also hold for non-linear SDEs such as the one proposed by Bartosh et al. (2024). We discuss the impact of the linearity of the drift in section 6/and provide the general formulas in the appendices. Anderson (1982) showed that there exists corresponding (sharing the distribution over trajectories) reverse-time SDE
with drift ft) d diffusion (t)
dæ: (s())az - (1) V log (x )dl d g())dW,,
(2)
2

---

Published as a conference paper at ICLR 2025
Reverse SDE (2)
Augmented PF ODE (3)
logar (ey)
Mawn the
May
Augmented Reverse (4)
[ng
(ag
Stochastic sampling
log estimation
e1 NFE
per step
wmm/
Aaun Moda
(2)
PF ODE (3 top)
Aug PF ODE (3)
/
x
Aug Rev SDE (4)
/
Figure 3: Tracking stochastic sampling likelihood. Estimation of log (ace) (colored trajectory) for stochastic sampling via Augmented Reverse SDE (equation[) Gaussian mixture with known Ve log P:()) and pr. Evaluation of d d'log Pr (a.,) requires only the score function.
where is the marginal density of the forward process (equation 1. and (We) is a Wiener process going backwards time. Training of diffusion models centers on efficient estimation the score V. log (ar) (Hyvärinen & Dayan 2005 Vincent 2011
Song al. 2020 Kingma & Gao 2024
2.1 DIFFUSION ODE
We are interested in sampling high-density regions of diffusion models and for that, we need ability to calculate the likelihood pu (a of sample image Pu- As the reverse-SDE (equation 2 does not track the likelihood evolution d log (x..). ). a typical method for obtaining sample likelihoods is instead via the Probability-Floo ODE (Chen et al. 2018 Song al. 2021c)
f(1)æ, -
-(1))=-10000 (() Va log (r,,)
Tt log (2x) = -(() (() div
dt,
V log (x(;))
where divæ Ee Dzc is the divergence operator. The PF-ODE continuous normalizing flow (CNF) that shares marginal distributions pr (ax,) with both forward (I) and reverse SDE ( when they share po (Song et al. 2021c). The PF-ODE tracks and returns both the sample and its likelihood log (2 from 2T ~ pr. requires second-order derivatives div which are roughly twice expensive the score evaluate (Hutchinson 1989 Grathwohl et al. 2019).
2.2 AUGMENTED STOCHASTIC DYNAMICS
has been reported that a stochastic sampler has superior sample quality to the PF-ODE (Song et al. 202 c Karras et al.) 2022). However, the absence of density-augmentte SDEs forbids
the stochastic sampler from describing the likelihood of the samples it yields (Song et al. 202 b
L1
al. 2022 Zheng et al. 2023 Lai al. 2023). We bridge this gap by generalizing equation account for the stochastic evolution of æ.
Theorem (Augmented reverse SDE). Let be a random process defined by equation
Then
1
I()): (() Va (og.)
In
d log pi (a:))
dt g(t)
dW,
f(t)D (t)l V (x;;)
V log (xx,)
(4)
The proof (Appendix consists of applying Itô's lemma and the Pokker-Plancl equation. Note that d log (x:) is economical track as it only needs access to the first-order score. Figure
3

---

Published as a conference paper at ICLR 2025
visualises particle following stochastic reverse trajectory while tracking its log-density. This result contributes a new useful and economical tool to generate a sample together with its density estimate, which previously done using PF-ODE (Jing et al. 2022 Similarly to the reverse SDE, we also introduce density-tracking forward SDE.
Theorem (Augmented forward SDE). Let be a random process defined by equation | Then
f(t)a, (11,
ID
[o
=
dt + g(t)
V logoog pa (2:c)
T dW.,
(5)
where
a(t = diva (s())x= - (t) Ve lo
59 (()) Vx px 1|
Proof similar the reverse case and can be found in AAppendix D The forward augmented SDE will prove useful for estimating the density of an arbitrary input point æ (not necessarily sampled from the model).
Karras et al. (2022) proposed more general forward SDE than equation for which we also derive the dynamics of log Pe ((x)) in both directions (Appendix F). Interestingly, we show that equation is the only case for which the dynamics of log (wx.) in the reverse direction do not involve any higher order derivatives of log (a+.).
We can now track the likelihood for forward and reverse SDEs under the theoretical true v log pe (xr). However, under score approximation s(t, æ) s v logpt(x) the density solutions log pr (xx) of Theorems and 2 become biased, which we study next.
2.3 APPROXIMATE REVERSE DYNAMICS
We can substitute the approximate score a(t, x) ~ Va (x) and assume pT 2ODE pt SDE N(0 Ip). The resulting SDE and ODE models are no longer equivalent, i.e. p ODE + pi Lu et al. 2022). The PF-ODE becomes
SDE
(ong et al. 2021b
- - (t) s(t, xz)
d
log Pt
ODE (a:))
Be
S(L)D (1) div s(1, 2+1)
dt,
which can track its marginal log-density log.2. ODE (x:)) exactly. However, substituting the true score
with st,, æ)) in the augmented reverse SDE incurs estimation error in log DDD (xrg) We charac-
terise the error formally in a novel theorem:
Theorem (Approximate Augmented Reverse SDE). Let s(t, x) be an approximation of the score function. Let Dr ~ pr and define auxiliary process starting at TT loog DE If
f(t)æt (t) s(t, xx))
ID
d
dl g(()
dW..
r+
-(1)D-
s(t, w.)
||2
s(t, xx))
then xu
po
SDE
and
ro log S (ax)) + X,
(8)
where X is a random variable such that the bias of ro is given by [g
T
EX
2 1.
(t)l s(t, xx)
Va log pt SDE
(xx)
1|3
2
(9)
See Appendix G for the proof and the definition of X. Intuitively, the true density evolution of S (x.) is induced by dæ, in equation 7 and the auxiliary variable r, does not follow perfectly. Since the equation 9 has an intractable score, we seek more practical alternatives to measuring the accuracy of ro in the next Section.
Pt
The new augmented SDE of equation 7 can be used with any score-based model without further tuning provide sample likelihood estimates log pg (x)o)) for no extra cost.
4

---

Published as a conference paper at ICLR 2025
3 ACCURACY OF THE DENSITY ESTIMATION
We analyse the accuracy of the log Po mate forward dynamics which will provide method for bounding the estimation error of ro-
SDE (xo) estimates in equation We begin with the approxi-
3.1 APPROXIMATE FORWARD DYNAMICS
In contrast to Theorem 3 when we replace the true score function with s in the forward direction, underestimate log Po ( on average.
(zo)
Theorem (Approximate Augmented Forward SDE), Lei s(t, xx) be the model approximating the score function and Do RD given. Define auxiliary process w starting at wg 0. If
f(t)æ
ID
) dt + g(t)
d W1 -
+ + (1) s(t, x4) 1|? + div s(t, xx))
dW..
21)
(10)
Then
WT log PT
(xT) - log (DE + Yo
(11)
where Yo is a random variable such that
T (
EY m0
(t) s(t, m3.)
log ps SDE (xt) 112 0. (12)
2
See Appendix for the proof and the definition of Y wo- Due the drift diy operator, the evaluation of dur is computationally comparable to dlogg Pt known lower bound into novel identity,
ODE
(xt).
Interestingly,
Theorem completes
Ani
dd
log Pb
SDE (ax)
EY
| 11 + B
l|o, stt, 0,20 + +,8))
=|
+ C, (13)
dt
20
BLBO( (Song, et al. 20216 Kingma et 202T]
where 1 ~~ ((,, e NO, In) and C 2 (1 + log (2ma²)) (see Corollary 1p. ELBO(x) is
the standard tool for estimating the SDE's model likelihood of an arbitrary point æ.
3.2 ESTIMATION BIAS log Po
SDE
Model optimized for likelhooo Model aptimized for sample quality
We are interested in using the stochastic sampler to obtain high-quality sample- density pairs (zo, log (zo)). We have now discussed two ways of estimating log DE with ro (equation 7 and
0.47bpd
0.22bpd
(1xc (0s)OrTT
ELBO(xo) (equation 13 Their estimation errors er (ro,
(2r)) and EELBO (zp) = log PD (p)) ELBO(#z) are intractable estimate due the presence of unknown score V Vogg pi (z.).
To log pi
R su 0.21bpd
R xe 0.099bpd
(xx) (xx)
205 0D5 (xn)
Figure 4: ro > log Pa relate strongly.
ODE (xo) > ELBO(xo) cor-
However, we can provide an upper bound on the bias of both estimators,
E D0n CeerrpPPEEC (a.) [FELBO (xo)]= + ((ao.ro) (ro, æo)]
E ((woro) ro ELBO (zo)
(14)
20 (equation 12
20 (equation
((s)(taaabble))
We can thus sample (mo, ro) using equation and the average difference To ELBO (ma) gives an upper bound on the bias of both ro and BLBO( This can be useful to assess the accuracy of both and ELBO( (o)) as density estimates for stochastic samples æo. Furthermore, we can estimate how much po
SDE differs from po ODEE by providing bounds for KL
SDE
[pg
Egg SDE (xo) EL)O log Pu
ODE (x)O)
< KL
PD
SDE ODE
[p6
(m
ro log Po
ODE (DE (z
(=) (tractable)
R2 (a) (tractable)
(15)
5

---

Published as a conference paper at ICLR 2025
R" and R' are novel practical tools for measuring the difference between ph and Pu D As demonstration, we train two versions of a diffusion model on CIFAR-I 10 (Kiizhessky et al. 2009). one with maximum likelihood training (Kingma al.) 2021; Song et al. 2021b) and one optimized
for sample quality (Kingma & Gao, 2024). Please see Appendix for implementation details. We
then generate 512 samples of (aro, ro) with equation and for each æn estimated log Po (mo))
using equation 2 and (L)O using equation 13
For both models we found very high correlations between To and log Po Surprisingly, it is the model optimized for sample quality that yielded both a higher correlation and lower values of R" (s) and R(s) suggesting a smaller difference between po SiD and Po ODDE and
OODE (xaa) at 0.996 and 0.999.
lower bias of log Po (xo)) estimation (Figure 4). We also estimated R (s)s), which was negative for both models which a trivial lower bound for KT.[ppBB/lng (DE) 0.
The density estimates ro log (D on log Pb PODE (xo) and correlate with it very strongly (> 0.99).
(mo) from equation empirically form upper bound
4 MODE ESTIMATION
High
Many diffusion models are trained by implicitly maxi- mizing weighted ELBO (Kingma & Gao 2024), and can be interpreted as (2024a) emphasized the role of likelihood by explaining the empirical success of guided diffusion models by their ability to limit low density samples Po (xo))) We explore this idea further by asking: what if we aim for samples with the highest ?
log p. (x,, X.)
models. Karras et al.
Low
Figure 5: Equation 16 accurately recov- ers the mode-tracking curve.
4.1 MODE-TRACKING CURVE
We first go back to assuming perfectly estimated score function V log Pt (az). p(arglaa))) denote the denoising distribution given by solving the reverse SDE (equation from æ, to zgg If we knew how to estimate the denoising mode yo (xx)) ) = arg max (( xt)) we could bias the sampler towards higher likelihood regions by first taking regular noisy samples æ, Pr (xxx) at various times t, and pushing them to deterministic modes (=()))
We approach this by asking seemingly more difficult question: ie. ys such that p(ys (x,)) = maxe, p(=,(x;) for all < t (See Figure 5 We show that whenever such smooth curve exists it is given by ODE:
find mode-tracking curve,
Theorem 5 (Mode-tracking ODE). Let (0,T) and æ, RD noisy sample. If there exists smooth curve s + ys such that px)) = maxe, p(x, (x)). then y: = Dz andfor s <
d
ds y, [(s)y= - (s) V, (ys) (s) A(s, y.)
V y Dy logp (y
(16)
2² where A(s,y) = (Vg (s) (5s)) = J/ セ and By = Ee
the Laplace
duf
operator. In particular:
p(yo xx.) max p (.))
(17)
The proof and the statement without assuming invertibility of A can be found in Appendix H We visualize on a mixture of ID Gaussians in Figure 5) how the solution of the mode tracking ODE (equation 16) correctly recovers the denoising mode curve, i.c. the mode of p(x, (x,) for all s < t.
4.2 HIGH-DENSITY SEEKING ODE
Using Theorem 5 in high- dinenssonli data is problematic. A smooth mode-tracking curve needs to exist, which is difficult to verify and need not hold (Appendix Jj. Moreover, equation [6] requires computing and inverting the Hessian and estimating third-order derivatives, which is prohibitively

---

Published as a conference paper at ICLR 2025
[ 0.95
{ = 0.7
t= 0.3
Figure 6: Equation 16 enables controlling the rrelihood-diversiie tradeoff. Blue dots are sam- ples from the data distribution, iee a mixture four Gaussians with the bottom left component having the highest weight. Orange samples generated with Algorithm
expensive (Meng al., 2021). Please see [Appendix K for more details. We propose an approxima- tion of equation by noting its high-order terms disappear under Gaussian data.
Remark (High-density ODE or HD-ODE). If po is Gaussian, then equation T6
becomes
dy. (f()) g²(s) V. log (og..(y.) ds,
(18)
Lee. the drift term of reverse SDE equation 2 If y/: æ, and equation 78 holds for < then
yo arg max p( xx)) O0e-amay
(19)
See Appendix for the proof. Even though equation 18 finds the mode of p( (z:)) only in the Gaussian case (for sufficiently small e -mmax),, we will empirically show that even for non- non-Gaussian data finds points with much higher likelihoods than regular samples.
In Algorithm [ we propose a novel high-density sam- pler that uses equation 18 bias the sampling towards higher likelihood regions. We choose a threshold time
Algorithm High density sampling
1: Input: Threshold E (0,7) Initial æT (O, ID)
t and sample x; (x.)), and then estimate (xx)) by solving equation 18 from
f to = O. Figure 6
3: Sample 2; (x
eq.
6 or
shows how the threshold controls the tradeoff between likelihood and diversity in toy mixture of Gaussians.
Yo < HDD-ODE(L,O, 0,)
eq. 18
5: Return
4.3 ESTIMATING MODE DENSITIES FOR REAL-WORLD DATA
We will next discuss how to evaluate the density of the high-density samples yo æ2 and regular samples I, empirically show that p(yo æx) p(æu æx) for æu (x))) For any æo, the denoising likelihood can be decomposed using Bayes' rule
log Poje (3o xx)) log Pejo (x, (z0) log loggo(()))))))))ooo)))))))))))) (x:t).
(20)
of (n)
For ag ~ p( (2)) we can use equation from (Iz, Tz = 0) to obtain No log Pu (xx)) - log (xx To estimate log poyololaa
could the PF-ODE, but that is inefficient! Instead, can obtain d ologgs Ps (ys) under HD-ODE with convenient, and to our knowledge novel, lemma:
Lemma (General instantaneous change of variables). Consider CNF given by dx; = ft, dt with prior Pr and marginal distributions pt, and a particle following some different dynamical system dz, = ft dt.. Then, if fi and f2 uniformly Lipschitz in the second argument and continuous in the first, we have:
d log (z(e)
div fi 21) (fe(t, 21) (i(t,
V= log)((z))
(21)
OOn of salving HD-ODE we would need to solve the augmented PF-ODE twice: once for ye from t to T and once for yo from to T. Instead, we perform one augmented 22 HD-ODE solve from & O.

---

Published as a conference paper at ICLR 2025
p(xolx)) log o()o/)))
High density Yo
Random denoised images Xy p(xo(xe)
Cr 0.134
0.991
0r 0.218
0.976
0, 0.345
0.939
Figure 7: Algorithm generates images with higher likelihoods than regular samples. For different noise levels i, compare the high density yo (equation (8) with random samples 2%g CIFAR-10. We find that the percentage of samples with higher likelihoods than yo O.
The proof in Appendix I.
When / = 12 we recover the standard formula Chen et al. (2018).
By setting fi =PF-ODE (3) and f2 =HD-ODE (88) augment the HD-ODE with its density evolution d dllog (w.)) ((s g9) div, V (v.))
( V ly.). ||2
(22)
ds
We trained a diffusion model on 11 and generated for different noise levels t and com- pared the high-density samples yo la, (Algorithm 1 against 512 regular samples æt ~ p( We found that p(yo (zx)) > p( ax:)) for all samples across different noise levels / (See Figure 7
Additionally, we compared the likelihoods of regular samples and the ones obtained with algorithm for different models, and values of the threshold parameter t. We found that algorithm samples have higher likelihoods than regular samples in all cases. For details, please refer to Appendix
HIGH-RESOLUTION DIFFUSION PROBABILITY LANDSCAPE
We demonstrated that algorithm 1 can generate im- ages with much higher likelihoods than regular sam- ples. However, after visually inspecting the high density samples Yo in Figure 7 we found that they correspond blurry images with much less detail than regular samples.
(low)
Bamne Freenge (x)
NLL (bpd)
2
To gain more insight we analyze high-density sam- ples on higher-resolution diffusion models We found that the samples with the highest likelihood were blurry images. Surprisingly, for higher values (, I, the samples were unnatural cartoons but still re- ceived higher likelihoods than regular samples. The training datasets do not contain any cartoon images. See Figure 9 A similar phenomenon occurs for latent diffusion models Appendix O
P 0.974
20
-00 (high)
20000 40000
80000
80000 100000 120000
png size
Figure 8: Density estimates correlate with information content (. jpg size)
Inspired by the empirical observation that the highest likelihood generated samples are blurry images performed the following two experiments. First, add different amounts of blur to FFHQ- 256 test images and measure the likelihood of the distorted image. We found that blurring always increases the likelihood and that the increase is proportional the strength of blurring (Figure 10).
WW used FFHQ-256 and Churches-256 models from gi tithub. com soanggsonn//coree_oo pytorch and ImageNet-64 fromlgi thgtthhbb.. com www./allabeeeeo
8

---

Published as a conference paper at ICLR 2025
Regular samples
0.57
0.37
NLL (bpd)
FFHQ-256 (Song 2020b)
Regular samples
1 0.57
1= 0.37
Church-256 (Song et al., 2020b)
Regular samples
0.01257'
== 0.052'
ImageNet-64 (Karras al., 2022)
Likelihood
Figure 9: Diffusion models assign highest likelihoods to unrealistic images. Left: High-density samples generated with Algorithm for various values the threshold parameter Right: distribu- tions of negative log-likelihood (in bits-per-dim). Across different models and datasets, Algorithm finds unnatural images, which have higher densities than regular samples.
Second, we compare the model's likelihood estimation with the image's file size after PNG com- pression. The smaller the PNG file size, the less information in the image. For FFHQ-256, found 97% correlation between log (*p) and the amount of information in an image (Figure 8). This hints why cartoons and blurry images have the highest densities (Appendix P). We used 192 samples for each of 7 blur strengths a E {0, 1, 2, 5, 10, 20, 50}. resulting in 7-192-1344 images.
6 DISCUSSION
Variance of log SDE SDE
Pú (x)) estimate. In section we discussed the accuracy of ro. novel esti-
mate of log Pü (zo). Specifically, we provided tools to estimate the bound its bias for stochastic samples. Based on the empirically measured correlation between To and log p0 (ODE(2)) at over 0.99,
NLL 2024
NLL 4845
ALL -0.00
HLL -2.00
NLL 10:00
(Be)
Guring Mering tinreehh
NLL -0
ALL -636
MLL
MLL 110
10
50 (high)
NLL (bod)
Figure 10: Blurring increases likelihood. Left: Two FFHQ-256 images with different amounts of blur and corresponding negative loglikelihoods (NLL). Right: Distributions NLL for different amounts of added blur (a E {0, 1, 10, 50), 192 samples each).

---

Published as a conference paper at ICLR 2025
hypothesize that the variance of ru log pt (z0) is low whenever st,, so V log ( (x)).
However, we do not provide theoretical guarantees, or empirical estimates.
ODE In equation 15 show that Rº = E [o log Pe [r)]] is upper PDDE vs Po
ODE
Po
bound on KL ODE (mn). However, for two different models, we found that ro 2 logr6 logroDe(c (x)))
ODE O. In particular, for (æo, ro) sampled with equation 7 we must have on
average ro 2 log Po
holds for every sample. We hypothesize this is a more widespread phenomenon. but do not prove it.
Towards exact estimates of log po ()DE(())) We showed in equation 9 that the bias of ro is given by E(ro - log po (2xo) x gr ((t] 2+) - V log p (x+))l for æ, ~ Pi SDD (,D))),) Similarly, equation 12 shows that log p (ao)) (
(a)) x ,x (1) {(1, x²2- V= log p. (a:)) (xx) for
at, ~ (2 Both these errors could then be reduced to zero if s(t, x)) V = log Pt (x)
(a) for all f, æ. However, for SDEs with linear drift (equation 1 this can only happen if yد ddD is Gaussian for
all & (Proposition B.1 in Lu et al. 2022)). This is because an SDE with linear drift cannot transform a non-Gaussian into a Gaussian in finite time T.
To unlock exact likelihood estimation in diffusion SDEs, non-linear drift is necessary, such as the one proposed in Bartosh et al. (2024). There it is possible to have pr Gaussian for finite T T'add s(t.x) = for all t, æ, in which case both ro and ELBÖ(x)) become exact (Theorem 7 and
Ve log pi SDE (x)
Proposition
7 RELATED WORK
Park et al. 2024) show that the temperature in the diffusion samplers relates to the sample density and can result in cartoon-like samples. See Appendixx Q for discussion on cartoon generation.
Likelihood estimation for diffusion models. discussed in |suseection 2.3 there is a distinc- tion between diffusion ODEs and SDEs. For diffusion SDEs, only lower bounds on likelihood are
reported (Ho et al., 2020 Vahdat et [202 Nichol Dhariwal 2021 Huang et al. 2021 Kingma et ala.. Kim et al. 2002). Exact likelihoods, on the other hand, are available for diffusion ODEs b. Dockhorn et al. 2022) and some works explicitly optimize for ODE likeli- hood (Lu et al. [2022] Zheng al. 2023 Lai et al. 2023). For a comprehensive survey, we refer to Yang et al. (202)). We provide a novel tool for estimating the likelihood of diffusion SDE samples.
(Song et al. 202
Concurrent work (Skreta et al., 2025) independently discovered Theorem I and Lemma I without
considering the forward process (Theorem 2) or imperfect score functions (subsection 2.3
Typicality likelihood. Theis et al.
(2016) observed that likelihoods do not correlate with image quality. Furthermore, deep generative models can assign higher likelihoods to nut-of-distribure
(OOD) data than training data (Choi et al. 2018 Nalisnick al., 2019a] Kirichenko et al., (020) and perform poorly at OOD detection. Nalisnick et al. (201b)); Choi et al. (2018) analyze this phenomenon with typicality, arguing that typical samples do not have the highest likelihoods. Ben- Hamu et al. (204)) observed that explicit distortion of an image, like inserting gray patch, increases the likelihood assigned by a flow-based model. Contrary to these reports, we explicitly study regions of highest likelihood and shed light on the probability landscape of diffusion models.
8 CONCLUSION
We provide novel tools for estimating the likelihood for Diffusion SDE samples. Additionally, we theoretically and empirically analyze the estimation error and discuss when exact likelihood estimation for diffusion SDEs might be possible. These tools, combined with a theoretical mode- seeking analysis, allowed us to study high-density regions of diffusion models. We made surprising observation that unnatural and blurry images occupy the highest-density regions of diffusion models. While Karras et al. (2024a) argued that avoiding low-density regions is crucial for the success of diffusion models, our analysis reveals that high-density regions should also be avoided in high- quality image generation. We discuss the limitations of this work in Apppendix R
This work not only enhances the understanding of diffusion model probability landscapes but also opens avenues for improved sample generation strategies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Finnish Center for Artificial Intelligence (FCAI) under Flagship R5 (award I 1501 1052). VG also acknowledges the support from Saab-Wad WASP (grant 41 1025), Academy of Finland (grant 342077), and the Jane and Aatos Erkko Foundation (grant 7001703). RK thanks Paulina Karczewska for her help with preparing figures.
REFERENCES
Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Ap- plications, 1982.
Grigory Bartosh, Dmitry Vetrov, and Christian A. Naesseth. Neural flow diffusion models: Learn- able forward process for improved diffusion modelling. In Advances in Neural Information Pro- cessing Systems, 2024.
Heli Ben- Ben-Hamu, Omri Puny, Itai Gat. Brian Karrer, Uriel Singer, and Yaron Lipman. D-flow: Differentiating through flows for controlled generation. International Conference on Machine Learning. 2024.
Jie Chen, Gang Liu, and Xin Chen. Animegan: novel lightweight gan for photo animation. In Artificial Intelligence Algorithms and Applications, 2020.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In International Conference on Learning Representations, 2022.
Adriaan Daniël Fokker. Die mittlere energie rotierender elektrischer dipole im strahlungsfeld. An- nalen der Physik, 1914.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenaud. Scalable reversible generative models with free-form continuous dynamics. In International Conference on Learning Representations, 2019.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. A variational perspective diffusion- based generative models and score matching. Advances in Neural Information Processing Sys- tems, 2021.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Slatistics-Simulatiin and Computation, 1989.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 2005.
Kiyosi ltô. On a formula concerning stochastic differentials. Nagoya Mathematical Journal, 1951.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional dif- fusion for molecular conformer generation. Advances in Neural Information Processing Systems, 2022.

---

Published as a conference paper at ICLR 2025
Hamidreza Kamkari, Brendan Leigh Ross, Rasa Hosseinzadeh, Jesse C. Cresswell, and Gabriel Loaiza-Gamee Ganem, A geometric view of data complexity: Efficient local intrinsic dimension estima- tion with diffusion models. In Advances in Neural Information Processing Systems, 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 2022.
Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding diffusion model with bad version of itself. Advances in Neural Information Processing Systems, 2024a.
Tero Karras, Miia Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyz- ing and improving the training dynamics of diffusion models. In Conference on Computer Vision and Pattern Recognition, 2024b.
Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and Il-Chul Moon. Max- imum likelihood training of implicit nonlinear diffusion model. Advances in Neural Information Processing Systems, 2022.
Diederik Kingma and Ruiqi Gao. Understanding diffusion objectives as the elbo with simple data augmentation. Advances Neural Information Processing Systems, 36, 2024.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in Neural Information Processing Systems, 2021.
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in Neural Information Processing Systems, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Er- mon. Fp-diffusion: Improving score- based diffusion models by enforcing the underlying score fokker-planck equation. In International Conference on Machine Learning, 2023.
Cheng Lu, Kaiwen Zheng, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Maximum likelihood training for score-based diffusion odes by high order denoising score matching. In International Conference on Machine Learning, 2022.
Chenlin Meng, Yang Song, Wenzhe Li, and Stefano Ermon. Estimating high order gradients of the data distribution by denoising. Advances in Neural Information Processing Systems, 2021.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In International Conference on Learning Representations, 2019a
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out- of-distribution inputs to deep generative models using typicality. arXiv, 2019b.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffiusion probabilistic models. In International Conference on Machine Learning, 2021.
Bernt Øksendal and Bernt Øksendal. Stochastic differential equations. Springer, 2003.
Chicago Y Park, Michael T McCann, Cristina Garcia-Cardona Brendt Wohlberg, and Ulugbek S Kamilov. Random walks with tweedie: A unified framework for diffusion models. arXiv, 2024.
VM Planck. Über einen satz der statistischen dynamik und seine erweiterung der quantentheorie. Sirzungberichte der, 1917.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Simo Särkkä and Arno Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
12

---

Published as a conference paper at ICLR 2025
Marta Skreta, Lazar Atanackovic, Joey Bose, Alexander Tong, and Kirill Neklyudov. The superposi- tion of diffusion models using the itô density estimator. In International Conference on Learning Representations, 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021a.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty Artificial Intelligence, 2020.
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in neural information processing systems, 34:1415- 1428, 2021b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021c.
Piotr Tempezyk, Rafal Michaluk, Lukasz Garncarek, Przemystaw Spurek, Jacek Tabor, and Adam Golinski. Lidl: Local intrinsic dimension estimation using approximate likelihood. Interna- tional Conference on Machine Learning, 2022.
L. Theis, A. van den Oord, and M. Bethge. A note the evaluation of generative models. In International Conference on Learning Representations, 2016.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 2021.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 2011.
Ling Yang. Zhilong Zhang, Yang Song. Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: comprehensive survey of methods and applications. ACM Computing Surveys, 2023.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in neural information processing systems, 14:16280-16291, 2021.
Jing Zhao, Heliang Zheng, Chaoyue Wang, Long Lan, Wanrong Huang, and Wenjing Yang. Null- text guidance in diffusion models secretly a cartoon-style creator. Proceedings of the 31st ACM International Conference on Multimedia, pp. 5143-5152, 2023.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference on Machine Learning, 2023.
13

---

Published as a conference paper at ICLR 2025
A
NOTATION AND ASSUMPTIONS
Table 1: Summary of notation and abbreviations.
Notation
Description
D
dimensionality the data
| = Σ, 27
Squared Euclidean norm of æ RD
Va, or ue
gradient w.r.t æ:
div= =E ōxi
8.
divergence operator w.r.t
(xc)
expectation [ (æ=)p(e))
TD
D-dimensional identity matrix
DKL(pl q))
Kullback- divergence = lEz~p (log log g(x))
C*
function has continuous derivatives of up order k:
82 Ax Ei
Laplace operator
Dz?
P0
data distribution
Pt
marginal distribution of process defined by Forward SDE equa- tion
p(ax, (at)) = Pajo (2 (acu)
Forward transition distribution. For lincar drift SDE (cquation (z x2)) = NN (ouaoo Ip
p(zulo (2:) = (200(())
Denoising distribution Poje (2ol2;)) = (2, (o / //p (æx)
SNR(() of
signal to noise ratio
A log SNR(1)
log signal noise ratio
Vx (o())(
"True" score function
a(t, m))
"approximate" score function
marginal distribution of process defined the approximate verse SDE equation
Pt
.DDB
marginal distribution process defined by the approximate ODE equation
W
Wiener process running forward time
W
Wiener process running backwards in time
In equation [] the drift term is linear in œ, which corresponds to most commonly used SDEs, because
it admits Gaussian transition kernels Pt|s for t (Sogg et al.) 2021c|bb Kingma al. 1021
Kingma & Gao 2024) and covers many common implementatrons of diffusion models (Ho et al. 2020 Song et al.
202 la Nichol & Dhariwal 2021
Dhariwal & Nichol 2021 Karras et al. 2024b
However, more general SDEs have been proposed that do not assume linear drift (Zhang Chen
2021 Bartosh al.
2024):
dæ, = ftt, x. dt + g())dW,
(23)
with f and g satisfying assumptions below. We define the approximate reverse SDE in the general case
dx; ..) - (1) xx)) dt g())dW.
(24)
In our theorems section and section 3, we do not assume the linearity of the drift and provide more general formulas. However, results in section only hold for linear drift SDE. We follow Song et al. (202Ib) and et al. (2022) and make the following assumptions in our proofs ensure existence of reverse-time SDEs and correctness of integration by parts.
14

---

Published as a conference paper at ICLR 2025
1. Po E C and Pa>polla>113 < cc.
2. V e [0.T] f(;,t) C². And 3C 0, E te [0, llf(t. C(++ [
3. 3C 0, Va. y E RD (ff - ytt, C|| yl22.
4. g and VT E [0, T), [ 0.
5. For any open bounded set O. 50' Jo
[p)p+ dt < co.
6. EC 0, Va E ER,, R [0, [Vpp [ ((L+))))))))) |=]))
7. 3C 0, Væ. y ERP
Vogg.: Cllar yll-
8. 30 > 0, = R2, t [0, [[([, c(1+
9. 30 > V, y E RP [)) s(t y) -0
10. Novikov's condition: E
exp S V lx - s(1, x)) (1d <0.
11. 14 E [0, 7,, 3k > 0 (x))), SDE ()) ODE (x) E O(e-leel))) as || + 00.
Additionally, we assume
12. Van E IRP: (xne(a))) (I (Z)II logr?
< co. where v(xo)) is the path
measure of equation starting at Zv.
13. E EEDDEDE (5o g v V log Pi SDE (xx) s(t, xz) x)) < co, where SDE is the path measure of equation 7
14. Vag E RR, Vt E [0, > holds for the linear drift SDE (equation I. where Pejo (2(/2)))) is Gaussian.
())()(/))) E O(e O(e-ll=11)) as ||ج||| + co. This trivially
B FOKKER PLANCK EQUATION
useful tool in some of the proofs is the Bokker-Plancl' equation (Fokker) 1914 Planck 1917
Øksendal & Øksendal 2003 Särkkä & Solin 2019). partial differential equation (PDE) governing
evolution of marginal density of diffusion process. For process described in Equation 23
the evolution of marginal density is described by the Fokker-Planck Equation:
i क. px(x) = div x(())));(x))) + g (t)
1
(25)
where Pt is the marginal density time i, which holds for all E (0, T) and x E RDD Equivalently,
1
log pxx)) div (5(1,x)) + 59 a)) (o)
at
(26)
Ve log [((t.x)
2 9 V logp(p)))))
1
C ITÔ'S LEMMA
The main tool for studying the dynamics of log-density of stochastic processes we will use in our proofs Itt's lemma (Ito] 1951), which states that for stochastic process
dx, xt)dt + g())
(27)
and smooth function h R R + R holds that
2 dt æ.) = - h(t,, x.) + h(t, x.) TT f(t, xx)) + 5 (1) dt g(t)
Ox hit,, x J.W,
Ox
(28)
more general version of Itô's 's lemma holds, which does not assume isotropic diffusion, but we do not need it in proofs.
15

---

Published as a conference paper at ICLR 2025
D
PROOF OF [HEOREM
Theorem (Augmented forward SDE). Let æ be a random process defined by equation 7 Then
[())x dt + g(t)
In
log (oz)
((t,
T dw.,
(5)
V log pr (xt)
where
F(t, div (f())a2 - (²) Ve log))
2 (t))
V (o.):( 11
For the non-linear drift (equation 23 we have (the difference highlighted in blue)
F(t, xz) = = (ftt, (t) V ((re) + 22 (t)ll V (og):(#:) ||
Proof.
We will apply Itô's lemma (Appeddix C h(t,z)) log):(>). From Equation 26 have
① h(t,z) div - (
[((t,z) 1 ( log pi (x)
(29)
and
δ htt, x.) f(t, f(t, + (
(t,, +
0t
dx
1 -diiva((t, + ( (EltIOT - " 2 log (rx)(()))
V logn(a) [e, x4) JT, 2 g (oggn)))))
div "9 ( Vo g
-diva (f(t,z)) V (t)l (
(30)
Finally, we get
d dlog (((r)) = -(- -div (-ddiva (f)) - -22 log))((;)) + 2 ( (ogm( dt
(31)
+ Ve log dW..
E
PROOF OF THEOREM
Theorem (Augmented reverse SDE). Let be a random process defined by equation
Then
1(1)a: (1) V (xx.)
Ip
d
æt log (2:)
dt g(t)
dW.
-((1)D (()) V log (x..)
1|2
V log Pa
(4)
For the non-linear drift (cquation 23 we have (the difference highlighted in blue)
I
[(t, (²) Va (o::)(x))
Ip
(oggi((ze)
dt dt+g(t)
dw.
div [(1, 2 (1)ll Va log
V (x
16

---

Published as a conference paper at ICLR 2025
Proof.
The reverse SDE Equation 2) can be equivalently written,
dæ. = (-S(T 1, wt) + (T - V log logpr-((xt)) dt + 9(T - t)dW,
(32)
=p((,a)))
for forward running brownian motion W and positive dt. We will apply Itô's lemma Appendix C
to g(,, x) = log (z) Since forward and reverse SDEs share marginals we can use Equation 29
i
ar 0(1 x) - - 1,, TT - -1) logpr-r(2)
(33)
1 g - V - log Ve log [((T t,
and
0
0
σi gt, xz) (t, xz)² plt, xx) + - g (tt
8x
div x)) 50 1A4 2 tog pr-
V Pog
f(T t,t,] 922 t) V logpr
(34)
loggo Pr (x,) (f(T--tro) t, g - t) V (og)
1 g 174 tog pr pr
d] (,, + 1 (T ()]l (xx))
Remarkably, the terms involving the higher order derivatives: Ae log log cancel out. Thus, we have
1 (T v log dt
dlogpr-((x.)
[ivv (f(( -t t,,x)) + + - (z))
(35)
which can equivalently be written as
d (x = ( (-div (t,t, xx)) - (t) logp((=))12) + g(t) log) TdW,,
(f
(36)
where W is running backwards in time and dl is negative.
F GENERAL SDES
Karras et al. 2022) showed that there is a more general SDE formulation than Equation 23 which
can interpreted as continuum between the PF-ODE and SDE formulations. Specifically, they showed that for any choice of B3 RR_ + R4 the SDE
dæ, = f(t,
xx)) f((t,z))
{ 3(t) Vt) log) dt + 223(t)g())WW
(37)
has the same marginals Equation 23 and it has reverse-time SDE
f(t, - (2 + (()) +(0)
(t) logp((z)) dt + / (2)())
dx:
(38)
One can therefore replace the original model (5 = 1)) with any choice of non-negative 3. We now derive the augmented dynamics of log (xr,) for any B3
17

---

Published as a conference paper at ICLR 2025
F.1 GENERAL FORWARD AUGMENTED DYNAMICS
Proposition For a: following Equation 37 we have
dlooge((r.) = (-divaJ 2 + 8(1) (2) logpt(a)))) +())) (000)//)))) lxgm((:)))?)) di
+ Vz log d..
(39)
Note that since B(t) 2 0, the higher order term Ar (a(;) is always non-zero for any B.
Proof. We proceed in the same way the proof of Theoremm 2 and apply Itô's lemma (Appendix Cj to ltt,, x) = log (a:). Since Equation 37 shares marginals with Equation 23 we can reuse the derivation of a h(1, x) from Equation 29
क7
1 ((, div (1) (ogm)))))))))) log))(>)) -(0 logp((a))
Ø1
Therefore for following Equation 37
2 dlog = 0t h(t,, xz) + V logp)(x))) [(ttzz) 2 - g² Vz log p(((s))
(og dl (23(I)g((V (og JoW,
-(-divaf div x.) + { S(t) 9 (t) logn( B(t)g (t)llV (tt)) logg( dt
+ 23((t)g(t) log (zg
(40)
F.2 GENERAL REVERSE AUGMENTED DYNAMICS
Proposition 2. For x following Equation
we have
1 2
dlog = ft, + ((t) gt()) logp:(x:) (())) logr:(>:)1l dt
+ V2
(41)
the only choice for which the higher order term Note that = 2 (corresponding to Equation involving Ag log (æ:) disappears.
Proof. Similarly to the proof of Theorem rewrite the general reverse SDE with positive dt and W going forward in time
da, (J(T- 1, (1 3(T -1) (T v) logpr-((2.)) di
(42)
+ 223(T' +) - t)dW,
18

---

Published as a conference paper at ICLR 2025
We apply Itô's lemma to gtt, log (og))-(()):
dg xz) gt, zi)t
Ot
-Vag(1, (1 -22
+ B(T +3T B(T g2 - logpT- di
+ - - () (l, sr)dtt
+ ( - t)g(T V log pr p dW,
f (T tt, + B(T - -t - 1
g² A) log log dt
+ 3(T - -T) +) log pr- (( 1|²dt
+ - - t) pr (x, dWW,
(43)
which we rewrite equivalently with dt < 0 and W running backward in time to obtain Equation 41
G APPROXIMATE MODEL DYNAMICS
Analogously to Theorem 2 and Theorem we can derive the dynamics of log p. (x))
Theorem 6 (Approximate augmented forward SDE). Let æ be random process defined by Equa-
tion 23 Then
d dlogr (ar) G(I, + Vn log SDE (x))(")
(44)
where
1 2 G(t, m)) div (f(t, 92 x)) (t))l x)112 - (1) log pt SDE (DE (x) - a²)
(45)
Proof.
Lu et al. 2022) showed that the corresponding forward SDE to Equation 24 is given by dx: (1(1, ax)) + (2) (V, log
Pt SDE (x::) s(1, ax))) di g())dW,
(46)
SDE (x) for x following Equation 23 We will apply Itô's lemma (Appendix C) to <tt,x) = log log D
(not Equation 46 which is intractable due to presence of Va log SDE). de hhtt x) can be evaluated using Equation 26
i
(tt,z) fiv (Ax log pt
(PF div. s(t,x))
29 (t) log (x)
Øt
V. logrr SDB [(1, + (1) (v logrt SDE (z) x))
/ 1 (1) log SDE (x:
1
-div 5 (t) log (
g² (t) div s(t,
v. log S (DB (z)²² (1(1, + 1 (1) log Pr
SDE (a:)
(47)
Therefore, we have
0
1
01 h((t, x) + Vah(t, (t)
diva. x)) + o2 div s(t
Va log SDE TT) (t) xx) , (1) log 1 (DE())))
-diva (f(t, x) -9 (2()s(( x)) + ( x)1 - logpi SDE (D)
- s(t, 122
(48)
19

---

Published as a conference paper at ICLR 2025
Thus for following Equation
we have
d log Pt (xx,) G(t, V log pt sDE (x,) dWW,,
(49)
where
(²()) (1) G(t, -ivv -g ( (t, x)| (t)l ( log pogripe
SDE (x) st, x)
(50)
Interestingly, Theorem 6 can be used derive lower bound for the likelihood of an individual data point Kingma et al. 2021 Song et al. (0211b.
Proposition 3 (ELBO for non-linear SDE). For any Do E RD and pt SDE marginal distribution of process defined by some ps and equation 24 for t T, have
T Br (t)lss(t x.) V. log log SDE (x,) +ELBO
log SDE (xO)
(51)
2
where 1 ~ (),, x. (rjo((r and
EL)O(xo) = oorr lam SDE (xT) TE, L(t
(52)
and L(t,z) = - = where one may choose any of the following L1, L2, La (one could also have different definitions depending on t):
Li x)) = div (f(t, x)) - gt) x))
(53)
L2(1, x) = (f(1, (1))) )² Vz, Pe Lz(1,x0= div div=x + 9 (1) a TV.. log logPejo (at, (20))
(54)
(55)
Proof: Using Equation 44
SDE (wT) '' G(t,x,)dtt L. g(t)V log SDE (x.) dWW,
log pú
SDE (xp) log S
(56)
for æ being a random trajectory following Eqqution 23 starting Tp. Using the definition of G(t, w):
log pD
SDE (mo)
log PT SDE (IT)
-iva (s(1, - 9² (1)s(1, + +22 ( (01) di
1 T
(57)
((t))]s(tt Va log pテ SDE (a:: )1=tt
2
v log SDE (ar))
We can take the expectation of both sides of Equation 57 w.r.t. x ~ v(0/22)) where =(x(2x)) is a path measure of starting To. Note that the LHS of Equation 57 is constant w.r.t v( 2n)) and
20

---

Published as a conference paper at ICLR 2025
thus it is equal to its expectation.
Bllog Po
SDE
(xo)
Eloog pT
SDE
(aT)
=log Pb
snn (ar)
"first
div (f(1, (())s(tt, ( + (t)lssss x;;11 dt
E
"socond term"
T
+ E
2 2.
(t))ss((t,))) - V Pt SDE (xx))) dt
"third term"
I/' v(()Vo (o:))
E
"fourth term"
(58)
First term.
Since the expectation is taken w.r.t (/)) we have
(e [lgg PT
SDE (xT) Pey-pri (n)) [log PT (xT)),
SDE
(59)
where Prio is the forward transition probability of equation 23 Second term
Using Fubini's 's theorem we have
1/'¹ ((t,z;)dt
1/" (Em) Ftt, xx)) dt
E E (mn)
(60)
1/' ((tt dt.
After substituting for F, we get
(er-pip(es Ft, x.) = (e (-diva - + + (t,))((
(61)
Note that for any t the divergence term under the expectation can equivalently be written in of three ways (integration by parts; assumptions and 14 in Appendix A):
E Ez æa) div (J(1, - ((()))s((
E Em ( (m))
[f(t, - (t))))(t,, az)) Va. log Pog
(62)
Em (mm
[- div (t, x.) 9² xV) logg(p(z5/200)
which holds due applying integration by parts either
(i) to f(t, mx) - (t) æ.) and p=)o(z)/zo) or
(ii) : s xx) and (ajo(aa
Third term.
1 T
(1)))s(((, v. logr SDE
SDE (x: 1
2 J
(63)
T
2 J. (t) (Eerrpiotas mo) (1s(t, - V. log SDE (xx))l dt
21

---

Published as a conference paper at ICLR 2025
Fourth term. Using Assumption 12 Appendix A and the fact that g(t) log li (x;) is W adapted, we have
/ g( log SDE
(xx)" (x:) TdW: = 0.
(64)
Combining all four terms yields the claim.
Corollary (ELBO for Linear SDE). For an SDE with linear drift (equation) 1 for any 2o E RD,,
assuming PT NN ID) we have
T Rr Eg (1) Vn log Pr SDE
log (DEE
(65)
where t ~ (),,T), ~~0, 2t = 04 + 0rE and
Amn
T Er,e (-
(-dAe)
dde
ELBO(xx) C
||00 2|
(- di
|la.((1, + ore) +ell²
(66)
and C --
1 (1 + log (2T0z))
Proof. In the linear SDE case (equation 1 we have (1)0(1))(zo) Naailou20, Using Proposition 3 we have
ELBO(xx)) -E Exp (er SDE (xT) + TEi, Ltt,
(67)
and we choose
1
x1), 29 (t) x)12 + (t)
3 (t)ls(( x)1 div (f(, + (t)s x) Vv log (x)
(68)
= =f((DD 20 (t)|l((t, - Va (z)=)) + - (+) (ogppp()/0()))
Since Va log (2)o( a:z0- and x, (1 x0) equivalent to æ; a + ore for of
NO0 Ip), we have v lo
= and ae
T ELBO(z,) = - ((t)
s(t, a,t + o,e) +
T
2
+ E
flog PT (xT)) + af, + Er (tt)
2
(69)
Br,e
(-ax)
|lo,s(t, a,20 a, a,20 + O1E) + +ell
di
dd.
+ Ee llog pT
SDE (ar + σTE)]
DTE, f(t)
2 dt
"first term"
"second term"
First term
D
Egg [log [oologp pT (ar + aree)
=
2
l)g
20
Ee Bellorap + orell
D
1
log(2mo7)
[xol + oz
(70)
2
20
D
Amin
(1 + loo (og(2
|
2
22

---

Published as a conference paper at ICLR 2025
Second term
1 Ld ((()
S
dlog oz
DTE,
D
dt D ologor log 00)
(71)
2 di
dl
Combining all the terms yields the claim.
We can now use Theorem to prove Theorem
Theorem (Approximate Augmented Forward SDE). Let s(t, xt) be the model approximating the score function and æo R given. Define auxiliary process wt starting at wa 0. If
[(t)aa
In
) dt g(t)
d
S(LD + +(() (2) (21l ax) 1| + div (11,
T
dW,,.
s(t, xt
(10)
Then
WT log PT (xT) (x:)) log Pb (x0o) Your
(11)
where Yao is a random variable such that
T 2 (o (t)ll 21)
EY 20
Vee log pł SDE (xt) 11 0.
(12)
Furthermore, Yap can be written as Yep Y1 Y2, where
2 {/
1
Y1
( p SDE (xt) s(t, dt
and
EY, = 0; Var Y2) = L'" (t) <pla (ma) [s xx) V. log pt SDE (x) dt.
For the non-linear drift (equation 23 we have (the difference highlighted in blue)
f(t,
In s(1, x.
d
Cr
b st. 1| + div s(t, x) )
dt+
T dWe.
Wi
- div xt) + (t)
Proof.
Using
Theorem 6 we have
SDE SDE (xo) log pT (xT) f T G(l,wi)dl- (' v(()V Mo poggrtEE ( dW
log ph
(72)
SDE log pt (xTT) {" dws + Y1 Y2.
where
1 ['
YI [1) log SDE x(1, (
(73)
2
and
Y2 l' g(t) (s(t, Va Logrr (x:)) dW..
(74)
Since 1e du,, = wp wi = wr: we have
logro
SDE (xo) log SDE (x)) WT Y
(75)
for Y Y, + Y2.
Similarly to Theorem we can derive the dynamics of log pt SDE (xx) under the approximate reverse SDE (Equation 24).
23

---

Published as a conference paper at ICLR 2025
Theorem 7 (Approximate augmented reverse SDE). Let be random process following Equa-
tion 24
then
Ia [(,, art) - (1) dt + g(t) V log pi SIDE (xx)² F(t,
d
2t logpi SDE (xx))
dW.,
(76)
(x)))
where
F(1, (0) -1---dvv div ((1,00) -g²()) (An log Pt
SDE (x) e
(x)
div s(1,
29 (1)) logy: SDE (x)
(x)]
=0 when Vt, log (
(77)
Proof. The approximate reverse SDE can equivalently be written
dx; = - (f - ( - - t, x;)) dt + g( - t)dW,
(78)
for dt and W running forward in time. We will apply Itô's lemma (Appendix C h(t, 2)) log SDE (z)) From Equation 47 we have:
0 lut,, x) = divæ f(T - t,xx
1 g 1) log log (x=((x)) - g²( div s(T' - tt
00
+ V (x)) ( 1, gg t)s(( - t, + 3
1 (T - V log SDE
log (x))
(79)
Therefore the drift of (t,t, x.2) is given by
0
L, mx) + (T - ()s(T 1,,,)) 1 55 (T 1),,,)(
0t (1, + (1, 0²)²²
(-J (-
ddva f(T - 1, x.) + (T 1o (o - g2 - 1) div 8(T - 1,,,.)
1
+ g +) log pp [1
(80)
and therefore for x following the approximate reverse SDE Equation 24), we have
dlogr pt SDE (x,) = Ftt, x; dt + g(t) V log SDE
SDE (x,)) dW,,
(81)
where < 0, W is running backwards in time and
F(t, div ftt,x) (t) log SDE (z) div stt, 2 g((()]] V log SDE (x)
(82)
L
(zt). However, d dloopr pÝ (P Theorem 7 defines the exact dynamics of log pi Vz log Pt StD (ax)), which we cannot access in practice. We only have access to the approxima- tion st. 2²). We now show that replacing the true V, log,;DDB with no longer provides exact
(æx) depends on
likelihood estimates, but an "upper bound in expectation".
Theorem (Approximate Augmented Reverse SDE). Let s(t, x)) be approximation of the score function. Let TT ~ Pr and define an auxiliary process r starting at rT = loo Sigr?po (xT). If
f(t)æ; - (t) s(t, x.)
ID s(t, xx.)
d
E2
dt g(t)
dW.,
s(1, w.) ||2
then Bo ~ Po
SDE (xx)) and
SDE (2t) + X, ro log Pu
(8)
where X is random variable such that the bias ro is given by
T
EX
2 ( (
(t) st,
Va log pr
SDF (xx)
1|2
2
(9)
24

---

Published as a conference paper at ICLR 2025
Furthermore, X can be written X X1 + X2, where
X1 = !'" (p))) div s(t,
1
5 [|s(t, x1)12 An log Pi
SDB (x)(),))))))))))))))))))
[ log SDB (x))) dt
p
2
and
/ g
EX2- = O; (2r(X2)
(1) xz) V log d SDE
(< 2
For the non-linear drift (equation 23 have (the difference highlighted in blue)
f(t, g²) s(t,
Id s(1,20))
d
æt
dt + g(t)
d..,
Tt
div 0 f(1, (1, - (t)) s(t, 1|2
Proof.
SDE (xo) = logtT SDE (xT) - 1-/ dlog SDE (xxe)
Po Pb
PT
(83)
From Theorem
we have
F(t, + S' v(t) SDE (x,) (Tx))dW, dWe SDE (xx) =
d dlog Pt
T
(-diva f(t, (()) dt
(84)
I'" g(t)s(t, xx) dw, + X1 + X2,
+
where
X1 {'' (t) (dive s(t, xx) [ts x1)1 A log Pt
1
SDE (xx)) (xx))) 2
Va log Pt
SDE
(x:) dt
(85)
and
X [ g(t) (V log SDE (a:)) - s(t, dW,
(86)
Note that from Assumption 13 (Appeddix A have
EX, 0.
(87)
Furthermore, using Fubini's theorem, we have
S' g (( (e))
EX1
(div s(t, + 2 |t xx)11
A log Pt SDE (x:)
(88)
マ log SDE (x:))1 dl.
Now rewrite
E (e.))) (diva st,, x2) D log p SDE
(DE(())) (xx))
E Bay (m:) div (s(1, x.)) - Va log SDE (x:))
(m.)
(89)
E E pm (w:) <(1 (x.) + V log p SDE (ax)))
V log SDE (x:))
E B (a:)) (-s(t, ) V log SDE (x:) + | x log Pt
SDE (x: l12)
where (i) is applying integration by parts (Assumptions and 11 Appendix A Substituting back to Equation 88 we get
EX, =
29 (t) D
E (DB (t) x22 (t, xx. " VVa logr; SDE (xx) + V log SDE (ax,) (x::)]1)
dt
g (1)EE (e [() - Va log ( 11dd
(90)
25

---

Published as a conference paper at ICLR 2025
Therefore
SDE (IT) - 1-² (-ddivaa f(t, - 1 (t)s(t, dt
log Po
SDE (xo) = log
PT
[(t,
(91)
L/e s(1)s(1, x.) dW,
where
T EX= /'
(1) B. (0) V logp; SDE (a: (x:
0.
(92)
H PROOF OF THEOREM
In the following sections, assume the linear drift SDE Equation In Theorem 5eee we explicitly assume Gaussian forward transition densities, which are only guaranteed in the linear drift SDE.
Theorem (Mode-tracking ODE), Let E (0, T) and a E RD a noisy sample. If there exists a smooth curve Ys such that ply, [x,) max=, p(z, xz)). then y: 2, for <t
1 gs) A(s ds y f(s)y= (s) V (U+)
d
V Vy logpo(ys)
(16)
0² where A(s.y) y) (V log ID)) (ss)= of e and By Ei operator. In particular:
et
the Laplace
ay?
p( (2))) = max p DD xz)).
(17)
20
Note that without assuming invertibility of A, Equation 16 becomes
A(s, ys) (v. - f(s)y, + (s) log - (5) (og.,))))
(93)
Proof. We begin by noting that for linear SDE (equation Pejs is Gaussian for < and therefore
log (og+)(())) = log Pr(a (z: y.) + log p. (y.) - log (xx:)
= |x.
f())u.1 1
(94)
(y.))(y)) log pt (æt);
(x)
where f(s) and (s) = of (s)o (See Appendix A.1 in Kingma et al. 2021 Since
(1) maxæ, (aje (2) (x) it must hold that
V log Po/e (3. = 0 for all <.t.
(95)
Therefore
d v,, logps)((u.1z:)) for all < t.
(96)
From Equation 94 we have
y. log Poje (0.) Vy. logp.(y.) + f(s)
{2(s) (m: --)(())))))))))))))())))))))))
(97)
and thus
(V y. log - ((s)x,)) 0.
(98)
ds
(2) (x)
i(s) (())
where (sss)
and (0s)
Note that
d V
ds v. (V+)
= as
Vy. log (ys) + V2 log ps (y.)8
(99)
26

---

Published as a conference paper at ICLR 2025
and we can use Equation 26 to re- werie the first term
a
Ōs V y. log PU))) = Vy.
Px
v. Ös
Vu. (-f(s)D+ 2 (s) (s) (Ve) Vy, ( 1 - V v. y logp, V (vs))
2 (s) log
-s(s)V log / log (v.)
2 -) Ay. log [(}) (V.)
- -V (u.) (s()) - V (og.
(100)
and thus
d Vy log 1 ,, logpo(us) fos)
Ay.
ds
2
(101)
V log (3+)) (ge- f(s)). + (0)s+
For the remaining terms we first note
f(s) - = exp(log a.) = expl expl/ L:
_ log au) = expl f f(u)}
d
(102)
O
du
and in particular de log [(s) = (ss). Similarly
o (²(s) = of (s)) of
o²
d e Au du
at
- = =
a
a²
du
= =at J." (-(Cuuu)
[ (-
dduu
- du = 0i I g'(u) du
c- su du = ai
(103)
du
du
a
a
= -f. Plu)g((u)du
=
and in particular d logg F:. _ (3) (s)g Therefore
d ds (-((5)y d())x)) = -s'() (65).. d((s)x,
(s)). d'()), (d(+))(()) s(s)((s)) y.
(104)
-s((s) (g. (g.-f(s)..) d(s)) x F(s)yz)
(s) (zi. f(s)y.) o(s) d ds (logø(s)) f(s)))))))
From Equation 97
have
d(s) x: -(())))) -V, log
(105)
and
d
(log (ss) d log f(s) ds
d
log (sg) +(s))
(106)
ds
ds
Thus
+ (()xx)) (-v()))++o())z)) = (ss) (g. f(s)g.) + - (())))) V, (y+))
ds
= --s(s) (j.- - f(s)y, +
+ f(s) log P. (y.).
(107)
27

---

Published as a conference paper at ICLR 2025
Putting it all together, we have
0 ds
(V logp(y,l<x.)
2 (8) (1l logpot))
-/ logpo
- (u0 ( log
(b(s)
1 (o logp.
(V) - (Ü. f(s)s,+
+ log
(108)
or equivalently for ( 3) v log I
1
y. = f()y, (s) (v.)
(109)
if (s,, ys) is invertible for all < and
v0()) = (s)
a
of
(s(s) am
oo (s))03
0?50
2
(110)
1
ex,
ex
1 ex
a x
o
e
Edi
o
ex
I MODE-SEEKING ODE IN THE GAUSSIAN CASE
We will prove the claims from Remark We recall for completeness. Remark (High-density ODE HD-ODE). Po is Gaussian, then equation T6 becomes
dy (f())y, (s) V log logps(y.)) ds,
(18)
Le. the drift term of reverse SDE equation 2 y:: B. and equation 18 holds for 8 I, then
yo arg max p (x,) + O(e -Amax) )
(19)
Ø0p
Proof. We first note that when po Gaussian and the SDE is linear (equation 1 then ps are Gaussian Vs. In particular V A log = for al E [0,T] and x E RDD Therefore equation 16
becomes equation 18
We will now study ys following equation 18 Recalling Equation 101
d ds
1
V log...(y.) , (s) log P (vs) ( (o.).
+ + 42 y Logpa((u (ji- + ( gf
= - -f(((s log (v.) + V logp. (1- f(s)y. + 9²) logp,
(111)
we then assume Equation 18
we have
V (y))) = - f(s) (8)))
(112)
ds
and to simplify further, using the fact that f(s) ds log a2
d (a, V. logP ds
d a, V, (v.p.
+ as d v, (og = 0. ds
(113)
ds
28

---

Published as a conference paper at ICLR 2025
Hence, for ys satisfying Equation 18 have
Og V y log Pa (3a) ar V (3)) for all < t.
(114)
We can thus rewrite
y. = f(8)y: - (s) V, lys)
f(s)y: g2(s) V. (y.)
(115)
(s) at V. (y))
O2
Furthermore
(2.0- g))) V log (e))) as yar
d
y/
a,
yea
ds
ax
a
(116)
g2( (s)
logpe(y))= dd, a V log
ag V
a
and we can solve
Vt
yo
di e
e a V logp(())) ds (f ed=e-duds at V (og))
Ogg
ag
J
ds
ds
(117)
(fm ed at V. (og): = =(e (e- e- -d ar V (v)))
Auaa
Leveraging that ag
1, we get
Y/
t log (3r) e Ann an V. log (z
Yo
a
+ of Va log (2:))
dnn Va (o:))((r))
(118)
=E [ [z]] - Aun a Va (og
arg maxp(xolos. (x)) + Amm).
NON-SMOOTH MODE-TRACKING CURVE
Distribution log p(x,(x.))
5" do
High
y,
log (X,,) XC))
8
Figure 11: The mode tracking curve need not be continuous. Left: mode-tracking curve with discontinuous jump at s* Right: distribution log (x)e(x,(x;) for values around s". At s* the arg max changes discontinuously.
29

---

Published as a conference paper at ICLR 2025
An important assumption in Theorem 5 is that for fixed (0, T) and noisy point 2; R?, there exists a smooth curve s + ys such that
Pste(ys maxps
(119)
It is assumption that need not hold. To demonstrate define the data distribution ID mixture of 3 gaussians p > W,N(,,, 02), where μ4 -2.5, #2 -1.5, 14 1 and o² 0.1 and weights w) = w/ = 0.274 and wi = 0.45. We model the distrbution with a VP-SDE (Song et al. 2021c), where of I+e 1 oz. We then choose at, 225 and such that X 8 and visualize log p(=,, for all < t and all 8t E [.5) and the mode-tracking curve y yx in white (Figure 11 leftt.
The mode-tracking curve exhibits a discontinuous jump at 8 such that Ag- Rs 1.28. The distribution (alo(ar (z.) has the mode at F, R 0.86 for s* ds and at Fs 2 -1.81 for = 8* ds
Figure 11 right).
K COST OF MODE-TRACKING
Evaluation of the drift of Equation 16 requires evaluating
(s,, ya) V A p. (y.))
Hessian factor
Laplacion factor
where A(s,y) 3)
og c cat and By Ei ga is the By? Laplace operator. We will discuss the factors separately assuming that we use model so x)) as Ve log (x)).
(V2 (og (s) (ss)
Hessian factor To evaluate A(s, need to estimate V log P. (3). which is the Jacobian ma- trix of the score function w.r.t. spatial argument y. This can be done using automatic differentiation and it requires D Jacobian-vector products (JVPs), where 1) is the dimensionality of the data and each JVP is roughly twice as expensive as score function evaluation (Meng et al. (021). In sum- mary, evaluating A(s, y=) requires roughly 2D score function evaluations plus the inversion of D x D matrix, which is ((D)).
Laplacian factor Evaluation of Vy log Vy div, Vy (os)s requires evaluating the gradient of the divergence of the score function. Exact evaluation of the divergence would again require D JVPs (Meng et al. 202T). However, one might approximate with single JVP using the Hutchinson's 's trick (Hutchinson 1989 Grathwohl et al. 2011)). One can thus approximate V,A, log ps (y.) V, div, V, log (y.) using a single JVP followed by a backward pass.
In summary. the bottleneck of the evaluation of A(s, ya) "ee'"VVVAA log Ps (y.) is the evaluation of A(s, ys)-i, which scales worse than linearly with the dimension of the data. For example, for CIFARIO 10 data, the evaluation of each step of Eqquaion 16 would be at least 6000x more expensive than the evaluation of the score function. For 256x256 images it would be roughly 400000x more expensive.
L PROOF OF LEMMA
Proof. From Equation 26 we have that
ot logp)(z) div, fi(t,z) V log f(ttt))
Therefore
d log Pr (zt) log Pi (z:) V loggp((z1) =
8
dt
= = dive (1, 2x) + Va log (12(1, - (1,
30

---

Published as a conference paper at ICLR 2025
M CIFAR MODELS HYPERPARAMETERS
In subsection 3.2 we train diffusion models on CIFARIO data. Specifically, these mod- els Variance Preserving (VP) SDEs with linear log-SNR noise schedule and g- parametrization (where the model is directly conditioned on À log SNR(t) as opposed to suggested by Kingma & Gao (2024)). Eø is parametrized wttp:///aapleleeeetteee
UNET using the im- with hyperparame-
sitss www.seeeeeeeeeeeeeeeeeeeeeeeeeee plementation from docs wwwgek..siteeeeeeeee....... ters: is.biggan=rue, dim_mults= (1, 2, 2, 2), hidden.size=
heads=8, dim head=16, 6, dropout rate=0.1, num res blocks=4, attn reso lutolutto ions= 16]; trained for 2M steps, 128 batch size, and the adaptive noise schedule from Kingma Gao (0244) with EMA weight 0.99.
The two model variants are:
CIFARIO-ML trained with maximum likelihood (ML), i.e. unweighted ELBO;
CIFARIO-SQ optimized for Sample Quality. i.e. trained with weighted ELBO with w(A)= = sigmoid( 2) as recommended by Kingma & Gao (2024).
N QUANTITATIVE ANALYSIS OF LIKELIHOODS OF SAMPLES GENERATED WITH ALGORITHM 1
In Table 2 we provide the values of |B|- log Po (2oo] (in bits-per-dim) for different models and sam- pling strategies. In all cases log (20) was estimated using the PF-ODE (Equation 3) ensure fair comparison. The values are mean + one standard deviation. We see that HD sampling (algo- rithm I generates samples with higher density (lower NLL) than regular samples across different models and values of the threshold parameter I. Note that for different models, values of the thresh- old parameter / HD sampling is different ranges. This is due to the fact that different models different SDEs and different noise schedules.
The models used are
CIFAR10 Models from subsection 3.2 with hyperparameters as defined in Appendix M Used 1024 samples for each sampling strategy.
ImageNet64 Checkpoint provided by Karras et al. (2022), i.e. Variance Exploding (VE) SDE with noise schedule satisfying t. "Original" sampling strategy is the stochastic Heun sampler proposed by the authors. Used 192 samples for each sampling strategy with default hyperparameters.
FFHQ256 and Church256 Checkpoints provided by Song et al. (2021c), i.e. VE SDE with exponential noise schedule. "Original" sampling strategy is the Predicoo--oorrect sampler recommended by the authors with default hyperparameters. Used 192 samples for each sampling strategy.
O STABLE DIFFUSION SAMPLES
In Figure we provide comparison of regular samples and high-density samples generated with algorithm using the Stable Diffusion v2.1 model (Rombach et al.) (222)) for multiple values of the threshold parameter t. Interestingly, even though the diffusion process happens the latent space, similar behavior pixel-space diffusion models discussed in section 5] Specifically, the high-density samples exhibit cartoon-lid features or are blurry images, depending on the threshold parameter value.
P WHY DO CARTOONS AND BLURRY IMAGES OCCUPY HIGH-DENSITY REGIONS?
Local intrinsic dimension (LID) a measure of an image's 's complexity and can be interpreted as "the number of local factors of variation" (Kamkari al. 2024). For image data, PNG compression
31

---

Published as a conference paper at ICLR 2025
Regular sample
HD(t = 0.2T)
HD(t = 0.AT)
HD(t = 0.67)
"Train"
" "Cat"
in a forest"
"Tent
Figure 12: Regular High-Density samples the Stable Diffusion v2.1 model. We added the prefix "A photo of" to each prompt to obtain more realistic images.
32

---

Published as a conference paper at ICLR 2025
Model
Sampling
NLL (bpd)
CIFARIO-MI.
PF-ODE
4.17 + 0.49
Rev-SDE
4.44 0.42
Rev-SDE
Theorem
4.30 = 0.41
HD(t 0.37)
2.65 0.62
HD(t 0.45T)
1.57 = 0.44
HD(L 0.57)
1.25 + 0.38
HD( 0.557)
0.98 = 0.36
HD(t = 0.7T)
0.24 + 0.27
CIFARIO-SQ
PF-ODE
4.55 + 0.46
Rev-SDE
4.23 + 0.42
Rev-SDE Theorem
4.16* = 0.42
HD(t = 0.37)
2.74 + 0.61
HD(t 0.45T)
1.61 = 0.39
HD(t 0.5T)
1.37 = 0.34
HD(! 0.557)
1.15 + 0.32
HD(t 0.7T)
0.46 + 0.30
ImageNet64
Original
3.16 + 0.81
Karras et al.
2022
HD(I 0.01257)
-2.10 0.33
HD(! 0.057)
-2.74 + 0.38
FFHQ256
Original
1.01 + 0.41
Song et
2021c
HD(t 0.5Z)
-2.58 0.65
HD(I 0.37)
-4.01 # 0.96
Church256
Original
0.77 + 0.23
(Song et al. 2021c
HD(t 0.52)
-0.66 + 0.34
HD(t 0.37)
-1.15 0.15
Table 2: Comparison of NLL (in bits-per-dim) for different models and sampling methods. soge denotes that the likelihood was estimated with Theorem I
size is commonly used as proxy for LID; that is, higher PNG compression size indicates higher LID (Kamkari et al.) (2024).
In Figure 8 we observed a strong correlation between the model's log- log-likelihood estimation and the
image's file size after PNG compression. This relationship can be attributed deeper connection between LID and the likelihood of data with varying levels of added noise (Tempezyk et al.) 2022)
Kamkari et al. 2024). Since diffusion models estimate densities across varying noise levels, their likelihood estimates naturally correlate with LID, providing intuitive explanation for the observed relationship.
This finding suggests that the diffusion model's (negative) log-likelihood estimates can be inter- preted as a measure the number local factors of variation. This insight helps explain why simple images, such cartoons blurry images, often exhibit higher likelihoods than complex, high-detail images.
Consider the samples generated with Stable Diffusion v2.1 in Figure 13 After zooming in, observe that high-density samples exhibit significantly less local detail and thus a lower intrinsic dimension.
33

---

Published as a conference paper at ICLR 2025
Regular sample
HD(t 0.47)
Regular sample
HD(+ 0.47)
Full image
Zoomed region
Figure 13: High-density samples have much less detail than regular samples and thus lower local intrinsic dimension.
Q CARTOON GENERATION
Recently, Zhao et al. 2023 proposed alter the generation in guided diffusion models bias the sampler to produce cartoon- like images. Specifically, the proposed method modifies classifier-fres guidance (Ho & Salimans 2022 by replacing the intermediate noisy model input corresponding to the null-guidance with the so-called "noise disturbance".
The main difference between our results, Zhao et (2023), and other cartoon-generation methods such as Chen et al. (2020). is that our aim was not to build a cartoon generator. Our goal was to study high-density regions of diffusion models and we developed method (algorithm [ to efficiently generate points from such regions. The fact that these samples turned out to exhibit cartoon like features was surprising discovery that we believe is of interest to a wider research community. Especially in light of recent report that inspired our study, which attributes the success of guided diffusion models to their ability to avoid low-density regions (Karras et al. (2024a). We show that targeting the highest possible densities is not desirable either in high-quality image generation tasks.
R
LIMITATIONS
Stochastic likelihood tracking While the likelihood estimation methods introduced in |section 2 and section apply to any diffusion SDE, regardless of how the score function parametrized, there are inherent limitations regarding stochastic sampling. Our novel method for likelihood esti- mation introduced Thehremem I jis beneficial as compared the PF-ODE (Equation 3) as it does not require estimating any higher-order derivatives and is free when doing stochastic sampling. How- ever, stochastic sampling usually requires more iterations than deterministic sampling (Song et al.] though it is roughly twice expensive to evaluate each step the aug-
2021c). Therefore, mented PF-ODE (Equation 3) as compared to the augmented reverse SDE (Equation 4). PF-ODE may require fewer total steps.
Mode-tracking
In Theorem 5) we derived an exact ODE, which follows the mode exactly. How- ever, there three limitations:
The result only holds for SDEs with a linear drift as the proof relies on Gaussian forward
transition probabilities;
Finding the exact mode is only guaranteed when smooth mode-tracking curve exists and it is difficult to verify practice and does not always hold (Appendix J):
The ODE is prohibitively expensive, especially in higher dimensions. See Appendix K
34


Reviews:
[{"review_id": "kVhJ5AO0uX", "reviewer": "Reviewer_NhXG", "summary": "This paper introduces new augmented SDE for likelihood estimation in SDE sampling of diffusion models. Moreover, a novel theoretical mode-tracking approach is proposed in order to locate the exact mode of the generative distribution and introduce a high-probability sampler capable of generating samples with higher likelihood than all the other samples, under theoretically proved guarantees. \nFinally, the main findings of the proposes analysis lies in the discovery of Cartoon-like images that lies in high-likelihood regions, even if the model has never been trained with images of the same style.", "questions": "- The related work does not take into consideration a very relevant paper \"Null-text Guidance in Diffusion Models is\nSecretly a Cartoon-style Creator\", Zhao et al which is not discussed. In particular it would be relevant to highlight some connection and insights with this previous related work. \n\n- See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and it develops new theoretical insights, especially the augmented SDE for tracking likelihood evolution are innovative and pratical since they don't introduce any additional computational cost.\n\n- The proposed high-probability sampler is a nice tool to generate high-likelihood samples that are not discoverable by traditional sampling techniques\n\n- The paper analize the diffusion probability landscape finding analyzing the different images in high-likelihood regions, experimentally demonstrating that the proposed theoretical tools can be used to perform analyisis of this landscape.", "weaknesses": "- The proposed mode-tracking approach has a very high computational cost and this is not clearly discussed. The discussion on how this mode-tracking approach scale and its computational limitations should be discussed and quantified\n\n- The high-likelihood samples discovered by the proposed analysis does not seem to have a real practical advantage, being cartoon drawings or blurry images. It is not discussed how these insight can be leveraged to improve sample generation strategies or how these high-density samples can be practically useful.\n\n- The limitations of the work are not discussed at all, a limitations section covering all the main potential limitations should be added.\n\n- The work does not report any implementation details, code and implementation are not submitted in the supplementary materials. It would be important to openly release the contribution upon acceptance to ensure reproducibility and improve transparency.\n\n- The analysis of high-density regions is done on small and simple diffusion models trained on restricted datasets with limited variability. Moreover the selected models and study is done on uncoditional sampling without any guidance from text. In my opinion it would be valuable to explore real-world and more complex diffusion models (such as SD, Flux etc), especially focusing on the impact of text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KxAePaNaSj", "reviewer": "Reviewer_p2yw", "summary": "This paper investigates the high-density regions of diffusion models, discovering that samples in these regions often appear as unrealistic cartoon-like or blurry images, despite the absence of such images in the training data. The authors propose a novel framework based on augmented stochastic differential equations (SDEs) to estimate the likelihoods of generated samples. This approach enables efficient, high-likelihood sampling without additional computational cost. The authors also introduce a high-probability sampling method that consistently yields higher-likelihood images than traditional sampling techniques, while their empirical analysis shows that images with less detail (e.g., blurred images) tend to achieve higher likelihood scores. The paper contributes to a better understanding of the diffusion model probability landscape and the relationship between likelihood and image quality.", "questions": "1. Does the high-likelihood sampling technique impact the diversity of generated samples, potentially leading to mode collapse or other reductions in output variability? \n\n2. For applications that prioritize both high likelihood and visual quality, what adjustments to the high-probability sampler could mitigate the production of low-quality images?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a novel framework for estimating likelihoods within diffusion models using augmented stochastic differential equations (SDEs) and high-probability samplers. This is an important advancement because it allows the exploration of high-likelihood regions without increasing computational costs. By deriving density estimates through augmented SDEs, the authors provide a theoretically efficient approach to analyzing model outputs across noise levels, setting a foundation for future studies in likelihood-based generative modeling. The findings reveal that high-likelihood samples often resemble cartoonish or blurry images—even though such images are not present in the training data.", "weaknesses": "1. Although the paper sheds light on high-density regions and likelihood estimation, it lacks a clear discussion on how these findings could practically inform the design or improvement of diffusion models in applied settings. Without recommendations on balancing high-likelihood sampling with image quality, it’s challenging to draw valuable insights from the findings, particularly for practitioners focused on real-world applications.\n\n2. The paper describes the emergence of cartoon-like images in high-likelihood samples but provides limited exploration into why or how this phenomenon occurs. \n\n3. The experiments are primarily conducted on well-known datasets like CIFAR-10 and FFHQ-256. Expanding the study to other types of diffusion models or additional data domains would strengthen the generalizability of the findings. Additionally, clarifying any specific architectural limitations of the proposed high-probability sampler could benefit those aiming to extend the approach.\n\n4. While the paper emphasizes the likelihood of generated samples, it does not address diversity within these high-likelihood outputs. The authors mention blurry and cartoon-like images, but they do not provide an assessment of diversity or how the high-likelihood sampling might affect the overall variance in generated outputs. It’s unclear if the method leads to mode collapse or reduces the richness of the model’s output space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYOaK4hqRS", "reviewer": "Reviewer_nSFN", "summary": "The authors offer a theoretical framework for estimating the log probability of samples that follow SDE in diffusion models. They develop novel forward and reverse augmented dynamics that estimates log probability not only for the case where $\\nabla \\log p_{t}(x)$ is known precisely but for the case where the score function is known. Then, the authors provide novel upper and lower bounds on the bias of the proposed estimator and analyze them on the experimental data. Lastly, the authors apply their proposed theory to analyze the diffusion probability landscape. They propose a simple yet effective way to generate samples with the highest log probability and show that such samples are unrealistic and blurry.", "questions": "1. Have the authors thought about how the proposed theory can be used to improve the quality of image generation?\n2. Can we use the proposed estimation for the log probability to evaluate model quality the same way as [1] (Table 2)? Will there be any difference between SDE and ODE estimation? Which one is better to compare different models?\n\n[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020c.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The authors develop a novel theory of augmented SDEs. They provided a clear and detailed derivation and coupled it with the bias estimation \n* Landscape analysis gives valuable insight into the structure of high-probability samples and provides a theoretical justification for the known fact that distorted images tend to have a higher likelihood \n* The paper is well-structured and easy to read", "weaknesses": "1. It isn't clear whether analysis from Section 5 can lead to the creation of better stochastic samplers or improve the quality of image generation. Overall practical implications of this work are quite poor \n2. There is no intuition behind observations from Figure 4. More precisely, what does it mean that the model optimized for sample quality yields a smaller difference between $p_{0}^{ODE}$ and $p_{0}^{SDE}$?\n3. Experiments were conducted on rather small and outdated diffusion models. It would greatly improve the scope of the work if the experiments were performed on the frontier models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Ef3nByCCn", "reviewer": "Reviewer_KEPG", "summary": "The paper presents a perspective on studying likelihoods for diffusion models. They introduce a stochastic way to keep track of the likelihood. They introduce a likelihood maximizing ODE that maximizes the likelihood of a path for any given starting condition. They attempt to argue that they can use this ODE to demonstrate that the high likelihood region of diffusion models results in cartoon like images.", "questions": "Please see weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is well written and the use of good figures and colors in equations really helps the presentation\n- They present a stochastic way to keep track of the likelihood and a likelihood maximizing ODE", "weaknesses": "- Up to section 4.2 everything was very reasonable and nicely introduced, however the use of the ODE in remark 1 for sampling is a big mistake. Real world data is multimodal, for instance CIFAR 10 has at least 10 modes. To use the assumption that the data is gaussian in order to use remark 1 results in incorrect results. Specifically this ODE doesn't sample from the correct target distribution, and therefore we observe those cartoon like images. \n- Additionally when comparing the likelihood of images sampled normally or from the HP-ODE the likelihoods were measured using different methods. In section 4.3 it is explained how they compute the likelihood of $p(y_0|x_t)$. Empirically they show that it results in higher likelihood than $p(x_0 |x_t)$, however since these likelihoods are being evaluated in different ways this comparison doesn't make sense. If done correctly the sample $y_0$ would be evaluated using the same way as $x_0$, so we would evaluate the likelihood of the **same distribution**\n- The paper has a limited amount of quantitative evidence\n- There is no explanation on the hyperparameters that were used during sampling", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)