Maximum Likelihood Training of Implicit Nonlinear Diffusion Model
Authors: Dongjun Kim
Byeonghu Na
Se Jung Kwon
Dongsoo Lee
Wanmo Kang
Il-chul Moon
Venue: NeurIPS 2022 Accept (2022)
Keywords: Diffusion Model
Score-based Model
Generative Model
Image Generation

Abstract: Whereas diverse variations of diffusion models exist, extending the linear diffusion into a nonlinear diffusion process is investigated by very few works. The nonlinearity effect has been hardly understood, but intuitively, there would be promising diffusion patterns to efficiently train the generative distribution towards the data distribution. This paper introduces a data-adaptive nonlinear diffusion process for score-based diffusion models. The proposed Implicit Nonlinear Diffusion Model (INDM) learns by combining a normalizing flow and a diffusion process. Specifically, INDM implicitly constructs a nonlinear diffusion on the data space by leveraging a linear diffusion on the latent space through a flow network. This flow network is key to forming a nonlinear diffusion, as the nonlinearity depends on the flow network. This flexible nonlinearity improves the learning curve of INDM to nearly Maximum Likelihood Estimation (MLE) against the non-MLE curve of DDPM++, which turns out to be an inflexible version of INDM with the flow fixed as an identity mapping. Also, the discretization of INDM shows the sampling robustness. In experiments, INDM achieves the state-of-the-art FID of 1.75 on CelebA. We release our code at https://github.com/byeonghu-na/INDM.

Maximum Likelihood Training of Implicit Nonlinear Diffusion Models
Dongjun Kim KAIST dongj Songjoou500kiis..co
Byeonghu Na" KAIST wp030520kaist.acc
Se Jung Kwon NAVER CLOVA
Dongsoo Lee NAVER CLOVA
Wanmo Kang
II-Chul Moon KAIST / Summary.AI
KAIST
Abstract
Whereas diverse variations of diffusion models exist, extending the linear diffusion into a nonlinear diffusion process investigated by very few works. The nonlin- earity effect has been hardly understood, but intuitively, there would be promising diffusion patterns to efficiently train the generative distribution towards the data distribution. This paper introduces data-adaptive nonlinear diffusion process for score-based diffusion models. The proposed Implicit Nonlinear Diffusion Model (INDM) learns by combining normalizing flow and diffusion process. Specifically, INDM implicitly constructs nonlinear diffusion on the data space by leveraging a linear diffusion on the latent space through flow network. This flow network is key to forming a nonlinear diffusion, as the nonlinearity depends on the flow network. This flexible nonlinearity improves the learning curve of INDM to nearly Maximum Likelihood Estimation (MLE) against the non-MLE ML curve DDPM++, which turns out to an inflexible version of INDM with the flow fixed an identity mapping. Also, the discretization of INDM shows the sampling robustness. experiments, INDM achieves the state-of-the-a FID 1,75 on CelebA. We release our code at https //ggttgt...////////////// com/ byw.noyu-nannnnnnnnn Na/NDM..
Introduction
Diffusion models have recently achieved success on the task of sample generation, and some works [1, 2] claim state-of oatte--f-th-attt theh-art performance over Generative Adversarial Networks (GAN) [3]. This success is highlighted particularly in likelihood-based models, including normalizing flows [4], autoregressive models [5], and Variational Auto-Encoders (VAE) [6]. Moreover, this success noteworthy because it is achieved merely using linear diffusion processes, such as Variance Exploding
(VE) Stochastic Differrnnial Equation (SDE) [7], and Variance Preserving (VP) SDE [8].
This paper extends linear diffusions of VE/VP SDEs to data-adaptive trainable nonlinear diffusion. To motivate the extension, though there structural similarities between diffusion models and VAEs, the inference part of linear diffusion process has not been trained while its counterpart of VAE (the encoder) has been trained. We introduce Implicit Nonlinear Diffusion Models (INDM) to train its forward SDE, the inference part in diffusion models. INDM constructs the nonlinearity of the data diffusion by transforming a linear latent diffusion back to the data space.
We implement the transformation between the data and latent spaces with a normalizing flow. The invertibility of the flow mapping is key to learning nonlinear inference part. Invertibility necessary
Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

SDE
(6x..) ((s,.))
Xo.1
Xa.2
Xo.3
Xa.4
Xo.5
No.6
Xo.7
Xo.8
Xa.9
Linear
Linear
Linear (VENP)
((5,.)
g(t)
Lineur
Nonlinear
C(x,,) = (1)
Noclinear
Semi- (Rotacing)
Figure 1: Examples of linear (top row) and nonlinear (middle/bottom rows) diffusion processes.
for constructing the nonlinearity, and we clarify this by comparing INDM with LSGM [9], latent diffusion model with VAE. Altogether. INDM provides the following advantages over the existing models.
INDM extends the scope of diffusion models from linear SDEs to implicit nonlinear SDEs.
INDM learns not only drift but volaility coefficients of the forward (inference) SDE.
INDM trains its network with Maximum Likelihood Estimation (MLE).
INDM is robust on the sampling discretization.
2 Preliminary
A diffusion model is constructed with bidirectional forward and reverse stochastic processes.
Forward and Reverse Diffusions A forward diffusion process diffuses input data variable, Xo ~ Pr. to a noise variable, and the corresponding reverse diffusion process [10] of this forward diffusion denoises a noise variable to regenerate the input variable. The forward diffusion is fully described by an SDE of dx; f(x,, 1) dt + Gxx,, f) dw,, and the corresponding reverse SDE becomes dx, = [xx,,t) div(GG") (x,, t) (GG")) (x,")xx t) log Pt (x:) dł + G(x,, t) dw, Here, W, E Id is an abstraction of random walk process with independent increments, where d is the data dimension, and dw, is the standard Wiener processes with backwards in time.
Generative Diffusion Having that the drift (f E R and the volatility (G E Rd terms are given a- priori, diffusion models [1] estimate the data score, Vx, log Ps (xx), with the score network, sø (x,, t). By plugging the score network in the data score, we obtain another diffusion process, called the generative SDE, described by dxe [f({ () div (x{, () (GG" (xx ()so
()] dī Gxx., t) dw; Starting from prior distribution of x ~ r and solving the SDE time backwards, Song et [1] construct the generative stochastic process of {xf T that perfectly reconstructs the reverse process of {x,)_nn under two conditions: 1) se (x,, = V log (x(x)) and 2) xT т. We
(xf,
define a generative distribution, Po, as the distribution of x
Score Estimation The diffuuson model estimates the data score with the score network by min- imizing the denoising score loss [1], given by [(xx)]Tnn X; = X(t) (lls (x),
xx, log (xx(Xx)xx)) (x)x/xx)]1 dt, where Poe (x; Xo) is transition probability of x, given Xo; and À is the weighting function that determines the level of contribution for each diffusion time. When G(x,, () -t), Song et al. [1]]. Huang et al. [12] proved that this loss with the likelihood weighting (A turns out to be a variational bound of the negative log- ligliikehhooo:: Ex, log po (xo)] < D( 92: Exy [log क (xr)), )]. up to a constant, see Appendix A. for a detailed discussion.
Choice of Drift (f) and Volatility (G) Terms The original diffusion model strictly limits the scope of diffusion process to be family of linear diffusions that f linear function of x, and G is an identity matrix multiplied by t-function. For instance, VESDE [1, 7] satisfies f = 0 with G = (t) drI and VPSDE [1, 8] satisfies f 2 (()xx, & X2 with G LB(t) Few concurrent works have extended linear diffusions to nonlinear diffusions by 1) applyng a latent diffusion using VAE in LSGM [9], 2) applying flow network to nonlinearize the drift term in DiffFlow [13], and 3) reformulating the diffusion model into a Schrodinger Bridge Problem (SBP)
[14-16]. We further analyze these approaches in Section 5.

---

Nonlingar Forward Path (data - noise)
Nonlinear Generative Path (noise - datal
16.0
X
X-
x
dx,"
Fo(x? :f)
dw
Axp
Fo
div div G
Go Vhone) of Go
Invertible Path (data - Intent)
Invertible Path
hg
h
hg
data)
2ij
(xp(Xo)
xff ng (x):
dxe
B(t)2 dt git) dws,
(00noaa +) of o(t) dia
+b 2
20
Linear Forward Path [latent noise)
RC π
Linear Generarive Parh [noise latert))
Figure 3: INDM attains a ladder structure between the data space and the latent space. The latent vector is visualized by normalizing the latent value, see Appendix F.5.2 for further visualization.
Motivation of Nonlinear Diffusion Process
Figure illustrates various diffusion processes on a spiral toy dataset. the top row, the diffusion path of VPSDE keeps its overall structure of the initial data manifold during the data deformation procedure to (0, I). The drift vector field illustrated Figure 2-(a) as black arrows presents that VPSDE linearly deforms its data distribution.
Unlike the linear diffusion, the middle row of Figure 1 with nonlinear drift shows that the
Usee Glodiggi
data is not linearly deformed (0, I). Fig- ure 2-(b) illustrates the corresponding vector field, in which distinctive components (or- ange/blue) are forced to separate each other. The nonlinearity of the drift term represented as rotating black arrows is the source of such nonlinear deformation at the intermediate steps, X0.2 Xo.6. When it comes to the volatility term, the last row of Figure presents the process with nonlinear G. Figure 2-(c) illustrates the covariance matrices the perturbation distribution at f with linear and nonlinear volatility terms, where the perturbation distribution induced by the volatility term is (0, G G) (x,.)) ') It shows the non-diagonal and data-dependent covariances of GGT in red ellipses of nonlinear volatility term, and the isotropic blue circles of linear diffusions.
(a) Linear x x2 (b) Nonlinear
(c) Nonlinear G
Figure 2: Vector fields various SDEs at 0.
Implicit Nonlinear Diffusion Model
There two ways to nonlinearize the drift and volatility coefficients in SDE: explicit and implicit parametrizations. While explicit is a straightforward way to model the nonlinearity, it becomes impractical particularly in the training procedure. Concretely, in each of the training iteration, the denoising loss L( L((x)) (xx)r== 0) requires 1) the perturbed samples X2 from por Par(Xe]e (x, Xx) and 2) the calculation of V log Poe (X, (x)). and these steps require long execution time because the transition probability Po: (x (x:(X)) is intractable for nonlinear diffusions in general. Therefore, we parametrize f. and G implicitly for fast and tractable optimization. As visualized Figure 3, we impose linear diffusion model on the latent space, and connect this latent variable with the data variable through normalizing flow. The nonlinear diffusion on the data space, then, is induced from the latent diffusion leveraged to the data space.
4.1 Data and Latent Diffusion Processes
Latent Difflusion Processes Let us define 2 to transformed latent variable z = (xo),, where h& is a transformation of the normalizing flow. Then, a forward linear diffusion
1 B dt + g(t) dw.,
(Latent Forward SDE)
dz
TTee covariance is dte Ex Ix, [Xeede [xe)(xe+ae -x: -x--fd)) (X", (x,,t).

---

starting at 2 = ha (x0) with Xo ~ Pr. describes the forward diffusion process on the latent space (blue diffusion path in Figure 3). The corresponding reverse latent diffusion is given by dz = - (()) - ( ot log pi (z) ) dt + g(t dw,, where p is the probability of z.
Forward Data Diffusion We have not defined the data diffusion process yet. We build the data diffusion from the latent diffusion and the normalizing flow. From the invertibility, we define random variables on the data space by transforming the latent linear diffusion back to the data space: x = h (z() for any / E [0, T]. Then, from the Ito's formula [17], the process {x{)]= follows
(Data Forward SDE)
dx (x
1) di + (x .t) dwr.
starting with x = he (zg) () From X hh (xo) Xo Pr. we call this process by induced diffusion that permeates the data variable on the data space. We emphasize that this induced diffusion collapses to linear diffusion if ha.-- id. See Appendix A.2 for details on drift and volatility terms.
Generative Data Diffusion A diffusion model estimates the forward latent score sø(2, 1) = v Flogp2 (z) with the score network, sg(z,t), to mimic the forward linear diffusion on the latent space. Then, the generative SDE on the latent space becomes
1 B(t)2) -9 (t())s((2), t) dē + d(t)(t)dd) dw:
(Latent Gen. SDE)
dz
2
with a starting variable 29 и. Thus, the process fx, xx generative data diffusion (purple path in Figure 3) with SDE of
=t= of xx-00 (x(() becomes
(Data Gen. SDE) dx."0
[fo- Gi) (G& G Vholso Voo (h (xx0)) dē G dw,
4.2 Model Training and Sampling
Likelihood Training Theorem 1 estimates Negative Evidence Lower Bound (NELBO) of Negaitve Log-Lik Likellikldood (NLL). For the notational simplicity, we define the targetted score function by
(Target of Score Estimation)
so ( t) = V log p. (2.))
Also, suppose C( ((24))
(t))) 9 2f of [||0 t) V log Por (z; 2 2 [[]]
dt, where
Pou (z, (zz))) is the transition probability of the latent forward diffusion. In Theorem I, we drop the constant terms that do not hurt the essence of the theorem to keep the simplicity. See full details and the proof in Appendix G.
Theorem 1. Suppose that Pœ, is the likelihood of a generative random variable x8.0 Then, the negative log- ligelikel is upper bounded by
Exx
log o (xx)) S (((xx) (gb,0))
where
1
C (( 01)
(²()) |Ilso t) sΦ (z, t)]l dt (xL(()()T) (1)
2
-Ex [og] det (Vha(xo))] + L [2 1r g²: 0) E, [log ( (2 (2)
Eq. (1) is the KL divergence DK ( (I where Ps and Vs.9 the path measures of the forward and generative diffusions on the data space. Eq. (1) explains the reasoning of why Sø is the target of the score estimation. However, the KL divergence is intractable, and Theorem I provides an equivalent tractable loss by Eq. (2), the summation of the flow loss with the denoising loss.
Algorithm 1 describes the line-by-line algorithm of INDM training. We obtain the flow loss by taking a flow evaluation. Afterward, we compute the denoising loss. We train the flow with Eq. (2). However, we train the score with L ((xx) \; (Φ, 0}) with various À settings for a better Fréchet Inception Distance (FID) [18].
Latent Sampling While either of red or purple path in Figure 3 could synthesize the samples, we choose the red path for the fast sampling (because the red path ford-forwards the flow only once), Starting from a pure noise zl ~ we denoise z to 2 by solving the generative process backward on the latent space. Then, we transform the fully denoised latent z to the data space 200
ho (z().

---

Table I: Comparison of INDM with previous works. N is the number of random variables.
Nonlineur Dillusion
Implemented
Laient Dillusion
Nonlineur F-Modeling
Nonlineur G-Modeling
Explicit f& Derived
Training
Sampling
Model
Data Diffusion
Complexity
Cost
+
X
Continoous
x
X
X
/
O(1)
LSGM
x
Continsous
X
x
O(1)
SBP
A
Discrete Discrete Continuous
x
Explicit Explicit Implicit
x
/
O
DiffFlow
x
O(N)
INDM
Continious
Implicit
/
O(1)
5
Related Work
Algorithm Implicit Nonlinear Diffusion Model
In this section, we compare INDM with previous works, and summarize our ar- guments in Table 1.
1: repeat
Get latent with flow by z hx(x)) for Xo Pr Compute -Ex [log] det (Vho(xa)
LSGM Vahdat et al. [9] put a linear dif-
4:
Get diffused latents {z, J with linear SDE
fusion on the latent space like INDM
5:
Compute (((2)) )T g 0) E 2 log π (z
but uses an auto-encoder structure.
6:
Compute flow loss C C Yi / 10, 0})
From this modeling choice, LSGM can-
aC,
not be categorized as a nonlinear diffu-
7:
Update & t Φ a6
sion model in a strict sense. Concretely, recall that a diffusion process is (mathe-
8:
Compute L 10 1 ); - [log ग. (2+)]
2
9:
Compute score loss C. = C [xx 1= x; ().
17
matically) defined as a sequence of ran- dom variables connected via a Markov
Update 0 < 0 11 ac,
10:
chain. From this definition, one needs to satisfy two requirements to call it a diffusion process: 1) there must be multiple (possibly infinite) numbers of random variables; 2) the random variables should be connected via a Markov chain. Unlike INDM, LSGM cannot build forward data variables from the forward latent variables because there is no exact inverse function of the encoder map, as long as the data dimension differs to the latent dimension (Lemma 3 of Appendix D.I). This leads that LSGM has no forward data diffusion process. From this point, analyzing the data nonlinearity becomes infeasible in LSGM.
11:
until converged
Moreover, LSGM has pair of key differences in its training. First, the latent dimension of LSGM is 40,080, which is 15x higher dimension than the data dimension (3,072) on CIFAR-I0 [19]. In contrast, INDM always keeps its latent dimension by the data dimension. See Table 9 to compare the latent dimensions of INDM with LSGM on benchmark datasets. Furthermore, LSGM is repeatedly reported [9, 20] for its training instability on the best FID setting of X a² (i.e., Lasmple [8]). Meanwhile, INDM is consistently stable for any of training configurations, see Table 8.
DiffFlow Zhang and Chen [13] explicitly model the drift term fa flow network, so the forward diffusion becomes dx, = fa dt + dw, However, there are differences between DiffFlow and INDM: 1) DiffFlow does not nonlinearize the volatility; 2) DifiFlow is too slow for its explicit parametrization (Table 18); 3) the flexibility of fo is too restricted; 4) DiftFlow has a larger loss variance (Table 10). See Appendix D.2 for the full details of our arguments. Focusing on the slow
training, observe that the denoising loss E xx,x [||so (xf; ()) V Poe (x, [xo) requires pair of
heavy computations: (A) sampling from x and (B) computation of V log Poe (x. (xo). Intractable transition probability Por (x, Xx) is the major bottleneck of the slow training.
To overcome the bottleneck, DiffFlow discretizes the continuous diffusion with N variables of discrete diffusion and uses the DDPM-style loss [8]. which does not need to calculate the transition probability. Under the discretization, however, the forward sampling of x takes O(N) flow evalua- tions for every network update. This sampling issue is an inevitable fundamental problem when we parametrize the coefficients explicitly. Having that the flow evaluation is generally more expensive than score evaluation given the same number of parameters, fast sampling is achievable only we reduce N. However, it hurts the flexibility of a diffusion process, so DiffFlow suffers from the trade-off between training speed and model flexibility. On the other hand, the training of INDM is invariant of N, and INDM is free from such a trade-off. Analogously, DiffFlow generates sample with the purple path Figure 3, so it takes ONN) flow evaluations, contrastive to INDM with single flow evaluation in its sampling with the red path.
5

---

SBP De Bortoli et al. [15] learn the diffusion process with a problem of pe DKL (pe pp). where P(pr, T) is the collection of path measure with pr and æ as its marginal distributions at t = 0 and T, respectively. It is a bi-constrained optimization problem search space that should satisfy boundary conditions at both & = 0 and = T. fa is the reference measure of a linear diffusion dx;- = f(x,, t) dt + (());));;; and the forward and reverse SDEs of Po are dx,- = [f(x,, 1) ( log (xx, 1)] d/++ g({) dw, and dx; = [[(x,, () - ()) log V 1] di- (t) dw,, respectively, where (V, 1b) is the solution of a coupled PDE, called Hopf-Cole transform
any path measure on the
[21]. Solving this coupled PDE is intractable, so the estimation target of SBP is V log V and V Vlog L As f and 9 are assumed to be linear functions, the nonlinearity of SBP is fully determined by (V, 1)
Analogous to DiftFlow, sampling from x, in SBP needs a long time. Few works [15, 16] detour this training issue using the ex- perience replay memory. Aside from the training time, the KL minimization puts the global optimal nonlinear diffusion pa near a neighborhood of the linear diffusion μ.. In other words, the op- timal pe- is the closest path measure on P(pr, T) to μ., so the inferred nonlinear diffusion would be the most linear diffusion on the space of P(pr, x). For the demonstration, we illustrate Figure 4: Norm Ratio of SBP. (t) V log (x,, t) l
Difuusion
in Figure We used the released checkpoint of SB-FBSDE [16], an algorithm for solving SBP, trained with VESDE on CIFAR-10. As = 0 in VESDE, this norm ratio measures how much nonlinearity is counted on a diffusion trajectory compared to the linear effect. Figure 4 shows that the ratio approaches zero except at the small range around ~ 0, meaning that the nonlinear effect is virtually ignorable than the linear effect. Therefore, Figure implies that the diffusion process is nearly linear in most of the diffusion time. We give detailed discussion of SBP in Appendix D.3.
6 Discussion
This section investigates characteristics of INDM. We show that INDM training is faster and nearly MLE in Section 6.1, and INDM sampling is robust on discretization step sizes in Section 6.2.
6.1 Benefit of INDM in Training
Having that DDPM++ is a collapsed INDM with a fixed identity transformation h& = id, the difference lies in whether to train & or not. This trainable nonlinearity provides the better optimization of INDM, as evidenced in Figure 5-(a), experimented on CIFAR-10 using VPSDE. shows a pair of critical characteristics of INDM training: 1) it is faster than DDPM++ training, and 2) it is asymptotically an MLE training. For the training speed, recall that the regression target of the score estimation is Sg. and this target is fixed in DDPM++ while keep moving in INDM. The target is constantly updated through the direction of sæ in Eq. (1) by optimizing S -5p-sell 11. attraction between Se and S& is what flow learning does in the optimization.
This bidirectional
For the MLE training, as the flow training is intricately entangled with the score training, we analyze INDM training for a specific class of score networks. First, we define S.oi (Definition in Appendix B) to be the class of forward score functions of linear diffusion with some initial distribution. Then, it turns out that it is the whole class of zero variational gap (=NLL-NELBO).
Theorem 2. Gap( Mo.0) == DKL ( (vs.e) - DxL = 0 if and only se E SSot-
Song al. [11] partially reveal the connection between the gap with Soot, by proving the if part of Theorem 2, in Theorem 2 of Song et al. [11] (see Lemma 2 in Appendix B). We completely characterize this connection by proving the only-if part in Theorem 2. Surprisingly, the variational gap is irrelevant to the flow parameters, and the MLE training of INDM implies that the score network nearby Soot throughout the training. Combining Theorem 2 with the global optimality analysis raises a qualitative discrepancy in the optimization of DDPM++ and INDM by Theorem 3.
Theorem 3. For any fixed so E Ssot 8 E arg DKL (1 then s (z, t)
8² V log Pi (z) =se(z,t) t), and DKI(HO V ee = -DKLpprlpes ,) Gap(ps Vo- 0) 0.

---

10t Steps
INDM DOPM
NELBO (Tea)
100k
Nikk Steps
LSGM
PD cpole imo)
INDM NLL-NELBE
5
NCSN+
NLL (Tae)
Teleanne RK48 ODE Solver
Disoratization
(a) Training Curve
(b) FID by Tolerance
(c) FID by Discretization
Figure 5: Comparison of INDM with baseline models, experimented on CIFAR-10.
Theorem 3 implies that there exists optimal flow that the forward and generative SDEs on the latent space coincide, for any score network in S the flow flexible enough. Therefore, INDM attains infinitely many (= ()) global optimums in its optimization space. On the other hand, DDPM++ has only a unique optimal score network, i.c., Sø- S Thus, Theorem potentially explains the faster convergence of INDM. We give a detailed analysis in Appendix B.
6.2 Benefit of INDM in Sampling
Figure 5-(b.c) equally illustrate that INDM more robust on the discretization step sizes in FID than DDPM++/NCSN++ To analyze the sample quality with respect to discretizations, recall that the Fuler-Maruyama discretization of the generative SDE (or called the reverse diffusion sampler, or simply the predictor [1]) iteratively updates the sample 2g with
- = Zi. + 77 (3 + + (sx))s(((
(o Zie tz) + g(tx)
until time reaches to zero, where Y/2 = 12 - 1--1 is the step size of the discretized sampler and
~ ",, I). The sampling error is the distributional discrepancy between the sample distribution of h (Zo) and the data distribution. Theorem 4 decomposes the sampling error with three factors: 1) the prior error Epri. 2) the discretization eтror Edis: and 3) the score error Eest- Note that Theorem a straightforward application of the analysis done by De Bortoli et al. [15] and Guth al. [22]. We omit regularity conditions to avoid unnecessary complications; see Appendix C.I.
Theorem 4 (De Bortoli et al. [15] and Guth et al. [22]). Assume that 1) sups, ||s@- (z, 1)
& (z) || < M, 2) sup2. || log pi (z)ll $ K, and 3) supz, [|0, log Pt (z) [z)l/ S Le-or log Pt
for some K. L, M, a > 0. Then
|p., (h5²) # Po S (pri(() + (aii(()) (Ø) + (ast 0) + +oV8+++
where Exprs (x) 22e T DRL DKI(P (1) (dis( 6v 6( (C /20)) and Eest ( 0) 2 with 5 = max 7/2 min
There are a pair of implications from Theorem 4.
V (d)i( and EEst(Ø 0) are independent of the discretiza- tion steps.
V ( V8 is the discretization sensitivity, entirely deter- mined by the latent distribution's 's smoothness.
Tolerance of faces Solver
To the deep understanding of the second implication, let us assume h&. (x) = ax for some scalar > 1, then the sensitivity is anti- proportional to a with the identical discretizations, i.e., Edis((a) ! FEss With smaller sensitivity of &.. there is more room reduce the number of discretization steps for Φa. Figure empirically supports the theory, showing that the sampler (at the large tolerance with 10 10-2) becomes more robust increases, CIFAR- 10.
Figure 6: Sensitivity analysis on scaled-up scenario.
Before we derive a concrete result from the implication, observe that the flow h& maximizing det(Vhg)) in Eq. (2). To understand the effect of flow training on the discretization sensitivity, let us restrict the hypothesis class of the transformation be linear mappings of h&, (x) ax. Then, as the determinant increases by the trained diffusion model would be insensitive to the discretizations. Now, for the general case, Figure 7-(a,b) illustrate || log (x) and || V log P (z)11/lzz| on
7

---

IDMM
INUM DDPM
Cosine Smiarity
DOPN
INDM
Difitsion
Cifiusia Time (1g)
Diflusion Time
(a) 2 log of (z)]l
(b) ||a. log (z)l//IZ)
(c) Cosine Similarity
Figure 7: (a,b) Comparison of INDM with DDPM+ for K, I,, (c) Cosine similarity of forward
diffusion trajectories on CIFAR-10.
CIFAR-10 10, respectively. Also, =| of INDM is slightly larger (1.3x) than DDPM++ There- fore, with these observations combined, we conclude that INDM is less sensitive to the discretization steps than DDPM++, from its loss design.
Second, the robustness could originate from the geometry of the diffusion trajectory.
The forward solution of VPSDE is x u(t)xo + v (-1-pp(t) with jut)
Table 2: Aver- age 73 Norm.
e 1 fu ((x) ds where the first term is a contraction mapping and the second term is the random perturbation. The contraction mapping points toward the origin, but the overall vector field of the diffusion path points outward because the prior manifold lies outside the data manifold, as shown in Table 2. This contrastive force leads the drift and volatility coefficients works in repulsive and raises highly nonlinear diffusion trajectory in DDPM++, see Figure 11 for a toy illustration. On the other hand, the flow mapping of INDM pushes the latent manifold outside the prior manifold, and the drift and volatility coefficients act coherently. Hence, INDM has the relatively linear diffusion path; see Figures 12, 13, and 14 for a quick intuition. Figure 7-(c) measures the cosine similarity of the ODE's diffusion trajectory with the straight line connecting the initial-final points of each trajectory. Figure 7-(c) implies that DDPM++ is under inefficient nonlinear trajectory that reverts backward near the end of the trajectory, as in Figure 11. In contrast, INDM trajectory relatively efficient and linear (Figure 15), which yields robust sampling by discretization steps; see Appendix C.2 for details.
Manifold
Norm
Data 776
Lasent
Price
3,072
7 Experiments
This section quantitatively analyzes suggested INDM on CIFAR-IO and CelebA 64 x 64. Throughout the experiments, we use NCSN++ with VESDE and DDPM+ with VPSDE [1] as the backbones of diffusion models, and a Resset-based flow model [23, 24] as the backbone of the flow model. See Appendix for experimental details. We experiment with pair of weighting functions for the score training. One is the likelihood weighting [11] with X(I) (1). and we denote INDM (NLL) for this weighing choice. The other the variance weighting [8] X(t) = o2(() with an emphasis on FID, and we denote INDM (FID) for this weighting choice.
We use either the Fredictor-Correate (PC) sampler [1] or a numerical ODE solver (RK45 [25]) of the probability flow ODE [1]. For better FID, we find the optimal signal-to-noise value (Table 14), sampling temperature (Table 15), and stopping time (Table 16). Moreover, sampling from z rather than æ improves FID because ()) collapses zero Theorem 4, see Appendix F.2.2.
We compute NLL/NELBO for performances of density estimation with Bits Per Dimension (BPD).
We compute NLL with the uniform dequantization, instead of the variational dequantization [26]
because it requires training an auxiliary network [11] only for the evaluation after the model training.
7.1 Correction on Likelihood Evaluation
continuous diffusion model truncates the time horizon from [0, 7] to T] to avoid training instability [27]. In the model evaluation, this positive truncation could potentially be the primary source poor evaluation (Figure I-(c) of Kim et al. [27]), so we fix = 10 5 as default in our training and evaluation. In the model evaluation, as the score network is untrained on [0, e). calculate NLL by the Right-Hane Hand--HaddSddee (RHS) of Eq. (3),
PM (x)(xx/xxx xx.) Poe (x, xn)
NLL = Exal l- log P5" (xo)] < Ex..x,
- log p. (xx) + log
(3)
8

---

Table 4: Performance comparison to linear/nonlinear diffusion models on CIFAR-10. We report the performance linear diffusions by training our Py PyTorch implementation based on Song et al. [1, 11] with identical hyperparameters and score networks on both linear/nonlinear diffusions to quantify the effect of nonlincarity in a fair setting. Boldface numbers represent the best performance in a column.
Nonlinear Data Diffision
NLL (1)
NELBO (L)
Gap (J)
FID (4)
SDE
Model
# Panms
after correction
before cornection
wi residual (after)
weo residual (bcfoec)
añt
(-NELBO NLL)
before
ODE
PC
VD
NCSN+ (FID)
INDM (FID)
63M
76M
4.86
3.22
3.13
3,66
4.89
3.28
4.45
3.24
0.06
0.03
0.79
0.11
2.29
2.38
DDPM-+ (FID)
X
62M
3.21
3.16
3.34
3.32
0.1
0.16
3,90
2,80
INDM (FID)
75M
3.17
3.11
3.25
3.18
0.86
0.07
361
290
VP
DOPM- (NLL)
62M
5008
2.97
3.13
3.11
0.10
0.14
6.70
5.17
(NLL)
7SM
2.98
2.95
2988
2.97
0.00
0.02
6.1
5.30
Table 5: Performance comparison on CIFAR- 10.
Table 6: Performance comparison on CelebA
SDB
Type
Model
#
Pierms
NLL
64 64.
NCSN+ (RID) [1]
DDPM++ (FID) [1]
10RM
3.19
2.64
Gap
Linear
DDPM++ (N]
108M
3.01
4.88
VDM [25]
1271
CLD-SGM [10]
10RM
2.25
DEGM [29]
2.92
Effieinn VIVAE [30)
1,83
FBSDE [16]
CR-NVAE [3]]
DemeeFoow-44100
(RID)
Stylehommer [32]
3.66
LSGM (NLL) [9]
269M
2.97
6.15
bused
LSGM (balanced)
LSGM (NLL) [9]
506M
109M
2.95
2,87
6.89
4.60
INDM (VE, FID)
NCSN++ (VE, FID)
2.31
3.41
2.33
3.42
0.02
0.01
2.54
3.95
Nonlinear
LSGM (balanced)
DDPM- (VP. FID)
2.14
2.21
0.07
2.32
Diffilww [13]
(VP, FID)
2.27
2.31
0.04
1.75
Flow
INDM (RID)
1JBM
3.09
2.28
DDPM- (VP. NLL)
2.00
2.09
0.09
3.95
INDM (NLL)
IND
0.00
INDM (NLL) +
75M
3.01
3.25
Here, th and p. are the model probability distributions at = 0 and = c. respectively; and P=)(-)x)) is the model's reconstruction probability of x given x., RHS of Eq. (3) is generic formula to compute NLL in continuous diffusion models, including DDPM++, LSGM, and INDM. Previous continuous models [1, 11, 9] have approximated Ex,, | log p5 (xo)] by EEx, | log (xo)].
There are two significant differences between our and the previous calculation: 1) the input of p. is replaced with X, from (Table 11); 2) the residual term of log (x x.] is added. With this (x.(xx|xx]
modification, our NLL differs from the previous NLL of Ex, [- log logg"" P.hh (Xo)] by about 0.03-0.06 in VPSDE, see Table 4. We report both previous/corrected ways in Table 4 and report corrected justrfiicaa
NLL/NELBO as default; see Appendix for theoretical justification of our NLL/NELBO corrections.
7.2 Quantitative Results on Image Generation
FID Boost with Pre-training Training INDM from scratch improves NLL with the sacrifice of FID compared to DDPM++ in Table 3. There- fore, we pre-train the score network by DDPM++ as default. This pre- training is intended to search the data nonlinearity near well-trained linear diffusions. Table 3 shows that training INDM after 500k of pre-training steps performs better than DDPM++ both NLL and FID. Appendix F3 conducts the ablation study of pre-training steps.
Table 3: Effect of Pre- training.
Model
NL.
FID
DDPM+
3,03
6.70
INDM (wl
298
(wll gra)
Effect of Flow Training Table 4 investigates how the flow training affects to performances, under the various linear diffusions and weighting functions. It compares the pre-trained NCSN++/DDPM++ with INDM, of which these pre-trained models initialize the score network of INDM. Experiments in Table 4 presents that INDM improves NELBO in any setting, and minimizes the variational gap to zero if we train the score network with the likelihood weighting.
SOTA on CelebA Tables and compare INDM baseline models. With the emphasis on FID, LSGM is the State-OF-The-Arr (SOTA) model on CIFAR- 10, but INDM- 18M (FID) is on par with LSGM-476M (FID). Moreover, we use Soft Truncation [27] to compare with LSGM (balanced). Soft Truncation softens the smallest diffusion time random variable in the training stage to boost sample performance by improving the score accuracy, particularly on large diffusion time. In the inference stage, Soft Truncation uses the fixed smallest diffusion time (e). INDM (NLL) + ST

---

1.00
0.0.1
122
133
106
FIR
8
S
S
S
S
Latens
Figure 8: INDM enables to learn diffusion bridge between two distinctive data distributions.
outperforms LSGM- 109M (balanced) in terms of FID with comparable NLL. Also, INDM- 121M (NLL) outperforms LSGM-2699 (NLL) in FID with identical NLLs. We achieve SOTA FID of 1.75 on CelebA in Table 6. See Appendix F.8 for an extended comparison and Appendix F.9 for samples.
7.3 Application Task: Dataset Interpolation
The linear SDEs fixedly perturb data variables, so such SDEs should have the end distribution pr (x(xr) as an casy-to-compute distribution. With the nomlinear extension, complex diffusion process exists transport /i another arbitrary p!?) However, common practice of diffusion models constrains only the starting variable by x ~ pr- so the ending variable of x is free to deviate from 2 Previous works have tackled this data interpolation task by using conditional diffusion model [33] or a couple of jointly trained source-and- and-target diffusion models [34] on paired datasets. Among
unconditional diffusion models using unpaired datasets, SBP [15] is natural approach for the task
by imposing bi-constraints with P(pr: πT) replaced by P(p!") (1) Pr (. ).
INDM can alternatively train the nonlinear dif-
V J Y h m b c C K U v t 9
U 6
3 7
2
(2) fusion from pfd) to pr with unpaired datasets.
INDM
0 7 &
7
O 4 E
T
7 2 S
7
We train the score and flow networks with a loss
i O 3 2 2 P N 9
0
0 2
(k..0 + (p,.(p,""111 (1pb):
W
t
C
R
0 &
LSGM
INDM NELBO
Interpolation Loss
~ e
3 7
G
X
14
Time t = 0.0
Time t= 0.5
Time t
1.0
where Ps is the probability distribution of x which is calculated by a single feed-forward computation of the flow network, see Algorithm 2 in Appendix F.2.3. The additional interpolation loss forces the diffusion bridge (x²*) TT to ahead towards pf by minimizing the KL divergence between x P6 and pf?). As the destined variable of the bridge becomes x h (zr) Rs (T), the flow network what constructs the interpolated bridge between couple of data variables, see Figures 8 and 9. Particularly, Figure shows that LSGM fails to interpolate letter to digit, and we attribute this failure to the non-existence of
Figure 9: Image-to-imas translation from EM-
NIST letters dataset to MNIST digits dataset.
(T),
diffusion bridge LSGM. Also, INDM models p(2)) with Ps.0 and p. with Pœ, so we could
(2)
compute NLL of each dataset separately: 1.10 BPD for MNIST and 1.56 BPD for EMNIST. In
contrast, SBP cannot separately estimate densities on each dataset. We emphasize that no additional neural network is needed for the interpolation task with INDM.
8 Conclusion
This paper expands the linear diffusion to trainable nonlinear diffusion by combining an invertible transformation and diffusion model, where the nonlinear diffusion learns the forward diffusion out of variational family of inference measures. A limitation of INDM lies in the training/evallutionn time. Potential risk from this work is the negative use of deep generative models, such as deepfake images.
Acknowledgements
This research was supported by AI Technology Development for Commonsense Extraction, Rea- soning, and Inference from Heterogeneous Data(IITP) funded by the Ministry of Science and ICT(2022-0-00077), Also, this work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea MOV))
10

---

References
[1] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations. 2020.
[2] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[3] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[4] Matej Greié, Ivan Grubišić, and Siniša Segvić. Densely connected normalizing flows. Advances in Neural Information Processing Systems, 34, 2021.
[5] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
[6] Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020.
[7] Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. Advances neural information processing systems, 33:12438-1244 2020.
[8] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 6851, 2020.
[9] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score- generative modeling in latent space. Advances Neural Information Processing Systems, 34, 2021.
[10] Brian DO Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326 1982.
[11] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances Neural Information Processing Systems, 34, 2021.
[12] Chin-Wei Huang, Jae Hyun Lim, and Aaron C Courville. variational perspective on diffusion- based generative models and score matching. Advances in Neural Information Processing Systems, 34, 2021.
[13] Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advances in Neural Informa- fion Processing Systems, 34, 2021.
[14] Francisco Vargas, Pierre Thodoroff, Austen Lamacraft, and Neil Lawrence. Solving schrödinger bridges via maximum likelihood. Entropy, 23(9):1 134, 2021.
[15] Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score-based generative modeling. Advances in Neural Information Processing Systems, 34, 2021.
[16] Tianrong Chen, Guan-Horng Liu, and Evangelos Theodorou. Likelihood training of schrödinger bridge using forward- backward SDEs theory. In International Conference on Learning Repre- sentations. 2022.
barwwrd-backward
[17] Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science Business Media, 2013.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.
2009.
11

---

[20] Tim Dockhomn Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically langevin diffusion. In International Conference on Learning Representations. 2022.
[21] Flavien Léger and Wuchen Li. Hopf transformation via generalized schrödinger bridge problem. Journal of Differential Equations, 274:788-8227 2021.
[22] Florentin Guth, Simon Coste, Valentin De Bortoli, and Stephane Mallat. Wavelet score-based generative modeling. Advances in Neural Information Processing Systems, 35, 2022.
[23] Ricky TQ Chen, Jens Behrmann, David Duvenaud, and Jömn-enrik Jacobsen. Residual flows for invertible generative modeling, Advances in Neural Information Processing Systems,
32, 2019.
[24] Xuezhe Ma, Xiang Kong, Shanghang Zhang, and Eduard H Hovy. Decoupling global and local representations via invertible generative flows. In International Conference on Learning Representations, 2021.
[25] John R Dormand and Peter J Prince. A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26, 1980.
[26] Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pages 2722-2730. PMLR, 2019.
[27] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and I1-Chul Moon. Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation. In International Conference on Machine Learning. pages 11201-11228,
PMLR, 2022.
[28] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, 2021.
[29] Eliya Nachmani, Robin San Roman, and Lior Wolf. Non gaussian denoising diffusion models. arXiv preprini 1rXiv:2106.07582, 2021.
[30] Louay Hazami, Rayhane Mama, and Ragavan Thurairatnam. Efficient-vdvae: Less is more. arXiv preprint arXiv:2203. 13751, 2022.
[31] Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34, 2021.
[32] Jeeseung Park and Younggeun Kim. Styleformer: Transformer based generative adversarial networks with style vector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 8983-8992, 2022.
[33] Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Flect, and Mohammad Norouzi. Palette: image-to-image diffusion models. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[34] Hiroshi Sasaki, Chris G Willcocks, and Toby P Breckon. Unit-ddpm: Unpaired image translation with denoising diffusion probabilistic models. arXiv preprint arXiv:2104.05335 2021.
[35] Simo Särkkä and Amo Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
[36] John Duchi. Lecture notes for statistics 31 3/llecctlical engineering 377. URL: https://stanfrdd edudclass/stats3l I/Lecturesffull notel. pdf. Last visited on, 2:23, 2016.
[37] Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1-34, 2011.
[38] Walter Rudin et al. Principles of mathematical analysis, volume 3. McGraw-hill New York, 1964.
12

---

[39] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[40] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. 2020.
[41] Erhard Glötzl and Oliver Richters. Helmholtz decomposition and rotation potentials in n- dimensional cartesian coordinates. arXiv preprint arXiv:2012.13157, 2020.
[42] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584, PMLR, 2020.
[43] Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics-Simulae and Computation, 18(3):0059- 1076, 1989.
[44] J. D. Hunter. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9 (3):90-95, 2007. doi: 10.1109/MSE.2000.55
[45] Avrim Blum, John Hopcroft, and Ravindran Kannan. Foundations of data science. Cambridge University Press, 2020.
[46] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
[47] Valentin Khrulkov and Ivan Oseledets. Understanding ddpm latent codes through optimal transport. arXiv preprint arXiv:2202.07477, 2022.
[48] Glen E Bredon. Topology and geometry, volume 139. Springer Science & Business Media, 2013.
[49] Desmond J Higham. An algorithmic introduction to numerical simulation of stochastic differen- tial equations. SIAM review, 43(3): 2001.
[50] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[51] Yongxin Chen, Tryphon Georgiou, and Michele Pavon. On the relation between optimal transport and schrödinger bridges: A stochastic control viewpoint. Journal of Optimization Theory and Applications, 169(2):671-691, 2016.
[52] Ludger Ruschendorf. Convergence of the iterative proportional fitting procedure. The Annals of Statistics, pages 1 1160-1174, 1995.
[53] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
2017.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[56) Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018
13

---

[57] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[58] Cheng Lu, Jianfei Chen, Chongxuan Li, Qiuhao Wang, and Jun Zhu. Implicit normalizing flows. In International Conference on Learning Representations, 2021.
[59] Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions, arXiv preprint arXiv: 2017.
[60] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[61] Lawrence F Shampine. Some practical runge-kutta formulas. Mathematics of computation, 46 (173):135-150, 1986.
[62] Alexia Combes. Adversarial score matching and improved sampling for image generation. In Interna- lional Conference on Learning Representations, 2020.
Rémi Piché-Taillefed Ioannis Mitliagkas, and Remi Tachet des
[63] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
[64] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties gan evaluation, 2022.
[65] Jean-David Benamou and Yann Brenier. A computational fluid mechanics solution to the monge-kantonne kanto--vnttooiich mass transfer problem. Numerische Mathematik, 84(3):375-395 2000.
[66] Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
[67] Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf, and Gert RG Lanckriet. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research, 11:1517-1561, 2010.
[68] Lawrence C Evans. Partial differential equations. Graduate studies in mathematics, 19(2).
1998.
[69] Valentin De Bortoli. Convergence of denoising diffusion models under the manifold hypothesis. arXiv preprint arXiv:2208.05331. 2022.
[70] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12104-121 14. Curran Associates, Inc., 2020.
[71] Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discriminator gradient flow. In International Conference Learning Representations, 2020.
[72] Yifan Jiang, Shiyu Chang, and Zhangyang Wang- Transgan: Two pure transformers make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.
[73] Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pages 1747-1756. PMLR, 2016.
[74] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
[75] Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. In International Conference on Machine Learning, pages 1660-1669, PMLR, 2020.
[76] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In International Conference on Learning Representations, 2020.
14

---

[77] Ali Razavi, Aaron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. In International Conference on Learning Representations, 2018.
[78] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu, Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 123-832, 2021.
[79] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We have reflected the contribution of this work claimed in the abstract and introduction in the main sections.
(b) Did you describe the limitations of your work? [Yes] See Section 7 and
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 8.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] I have read the ethics review guidelines and ensured that our paper conforms them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Appendix B, C, and G.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix G.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] We have released our code and instructions at https: htttpp:////a /ttg///tttttbttt www.ttt......yo..eeeeeeetetettet woom/byeonghuuna//a1
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix F.
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] Figures 4, 7-(c), and 19 report the error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix F.4.
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix F.
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material as URL? [Yes] We have released the asset on CelebA pool-3 features at nttps://githun com/ byeonghu- Da/INDM.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] Our asset is representative asset in the field of diffusion models. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "v1TEdA8svTV", "reviewer": "Reviewer_19AU", "summary": "This paper proposes to combine normalizing flows and denoising diffusion models in which the data distribution is first transformed into a latent space by normalizing flows and then the latent distribution is modeled by generative diffusion models. By leveraging Ito's theorem, it is proved that the resulting diffusion process is non-linear in the original data space. Therefore, the authors call it implicit nonlinear diffusion models (INDM). Experimentally, the proposed INDM achieves zero variational gap and state-of-the-art FID on the CelebA dataset.", "questions": "I like figure 9. It tells me to some extent what is learned in the latent space. Could you provide more cases like that, say the results on the datasets like two moons, pinwheel, checkerboard ... Also, an illustration of Cifar and CelebA is also important. From figure 3, it seems that the flow models just learn some color transformation. In this regard, the flow models make tiny contributions to data fitting. Could you show more cases on that?", "limitations": "\n- The model archiecture is similar to LSGM and the benefit of flows is unclear.\n\n- The results look convincing enough. Though, the onerous discussion make me hesitate about its practical value. The authors should explain their approach in simple and concise language.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe application of Ito's theorem is exactly interesting. It shows a strong connection between diffusion models and normalizing flows. And I think this connection is intuitive since the diffusion models can be considered a special case of flow models. Moreover, the reduction of the variational gap is also intriguing by applying normalizing flows.\n\n\n**Weaknesses**\n\n- The benefit of normalizing flows is unclear. Although the authors try to explain by the means of optimal transport and relative energy, it still lacks an intuitive understanding of why flows+diffusion makes sense, say does the transformed latent space make it easier for diffusion models to learn? One virtue of normalizing flows is to transform complex data distribution into a simple form, then one could learn a model on the simple manifold. However, it is unclear what is learned by flow models in INDM and how flow models increase the efficiency of learning and sampling.\n\n- It seems that the authors try to hind something. I think the main contribution of this paper is to show that the combination of normalizing flows can improve the expressivity of diffusion models. However, the authors do not discuss the benefit of normalizing flows thoroughly, but use a lot of vague discussions to make readers suck.\n\n    - Sec6.1 shows a better converge on NLL and NELBO. But as mentioned before, it's unclear why the normalizing flow benefits this. The author tries to contribute this to the \"bidirectional attraction\". But after reading the main paper, I can't understand how the flow models can be used to reduce the variational gaps, say NLL=NELBO, and make the learning curve of INDM more MLE.\n    \n    - Sec6.2 shows the diffusion process of INDM is nonlinear. However, I don't think this illustration deserves to be shown in the main paper because Ito's theorem has told us everything. The appearance of sec6.2 just traps the reader in the details of the eigenvalues of the covariance matrix.\n\n    - Sec6.3 extremely distracts my focus. I don't think the connection of optimal transport makes good contribute to the INDM. I must admit that I simply ran out of energy and gave up on close reading by the time that I got to the description of optimal transport in section 6.3. It is okay to show this connection, although it is still intuitive as I understand. But it is worth appearing in the appendix. Understanding this part in the main paper just makes readers feel distracted.\n\n    - It seems that the relative energy is a good way to show the benefit of normalizing flows. This part deserves further discussion. However, I find the discussion on it is also quite vague. To be honest, I cannot fully understand smaller relative energy contributes to faster sampling. I think the authors should explain this in simple and concise language and give only the necessary supporting evidence for their claims.\n\n- INDM seems an incremental improvement of LSGM. The diffusion and generative processes are similar. The only difference lies in that LSGM is based on the car but INDM is based on the normalizing flows. Just push back, the benefit of learning diffusion models on the latent space transported by flows is unclear, considering that the benefit of learning on latent space of LSGM is apparent, which learns an expressive prior for VAE and simple latent space for diffusion models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oiU1zZ8WU9g", "reviewer": "Reviewer_Aizs", "summary": "The paper presents a trainable non-linear diffusion process as a generative models. This work is an expansion of previous successful works that use linear diffusion processes and showed state-of-the-art results on sample generation.\nThis non-linearity is applied implicitly, helping avoid the alternative of an explicit non-linearity in the drift and diffusion coefficients, which leads to an intractable probability term that causes long training time.\nThe key idea of the presented method is to transform each data point to a latent variable using a non-linear invertible transformation, and applying a linear diffusion process on that latent variable, where in each point in time one can retrieve the relevant diffused image from a diffused latent variable.\nThe authors showed that their method also constructs a generative process that is easy to sample from.", "questions": "See my comments in the Weaknesses section.\n", "limitations": "The authors have addressed their method's limitations.\nThe described work has no potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\t- The problem is known to be valid and challenging, and the proposed idea is novel.\n\t- The method and the idea behind it are well explained in the paper.\n\t- The authors provide a motivation for non-linear diffusion process by presenting very nice figures of diffused data and its vector field during diffusion, that shows a clear advantage of non-linear diffusion over the linear version.\n\nWeaknesses:\n\t- The quantitative results are not the best in all , but the gaps between them and the SOTA results are not too big, especially when considering the number of parameters required for the training.\nIt would be nice to see qualitative results on the CelebA dataset, as the gap in FID seems to be significant. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iAAAJ5yseT", "reviewer": "Reviewer_KAbZ", "summary": "The authors expend the linear diffusions of SDEs to a data-adaptive trainable nonlinear diffusion. They propose a new model, implicit Nonlinear Diffusion Models (INDM) to train its forward diffusion SDE. The proposed model constructs nonlinearity of data diffusion by transforming a linear diffusion process on the latent space back to the data space, where the transformation is implemented with a normalizing flow.", "questions": "Are there other generative models that integrate normalizing flow to be considered as baselines in the experiments?", "limitations": "1. The motivations of integrating normalizing flow into the diffusion model is not convinced. Can we integrate other component rather than normalizing flow into the diffusion model?\n2. The motivations of why invertibility is important is not clear, although the authors provide a figure and tried to illustrate this.\n3. Normalizing flow has been widely used in generative model such as normalizing flow-based VAE. Thus, the idea of integrating normalizing flow into diffusion model is not novel, as diffusion model is categorized into generative model.\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Nonlinear diffusion is taken into account in diffusion models.\n2. The proposed diffusion model is able to learn forward diffusion process out of the variational family of inference path measures, and the corresponding optimization is able to provide MLE training and a good diffusion bridge.\n3. Experiments are carefully conducted and analyzed.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KuLpZvzW45h", "reviewer": "Reviewer_befw", "summary": "The authors propose a variant of latent diffusion models using invertible flows, this enables likelihood-based training. By training a linear diffusion model in the latent space, the corresponding diffusion model in the data space will be nonlinear. Training the flow allows one to learn both drift and diffusion volatility in the forward data-space diffusion.\n\nThe authors demonstrate excellent generative performance on standard datasets, as measured by FID scores and show how the method may be adapted for interpolation.", "questions": "- It is not clear what figure 7 is showing, shouldn't the end points lie along the same support for each method in data space if the diffusion model is working?\n- Does the interpolation loss require taking a gradient through time in the diffusion simulation? If so what is the memory requirement, does this break for larger images?\n- See questions on SB in weaknesses above\n- Is the extensive pretraining regime required? \n- What is the total train time including pretraining vs other baselines (without pretraining)? \n- The latent is the same dimension as the data, is there any computational savings?", "limitations": "The authors briefly comment on training time in the final section, however, do not detail the extensive pretraining regime required.\n\nIt would benefit the paper to address some issues in the discussion regarding OT, and move this to the appendices, then instead spend more time discussing the training procedure, provide an algorithm and discuss the importance of the pretraining regime which will be of significant importance to practitioners.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n- The authors propose an elegant method to train latent diffusion models using flows as encoder/ decoder, this permits efficient training of nonlinear diffusions via linear diffusion in the latent space and gives an explicit nonlinear diffusion in the data space\n- The method exhibits excellent generative performance measured by FID scores \n\nWeaknesses:\n- The paper would benefit from an algorithm detailing the exact training procedure\n- In the appendices, line 599, it is stated that the diffusion model is trained without a flow (for 5 days), then is fine-tuned with the flow. This is a very important detail and should be included in the main text. Does the proposed method not work without pretraining? The authors should include the performance for the diffusion without the flow first to indicate what the benefit is. Similarly, given 5 days is a significant amount of time would this not mean the training cost is a lot more than regular diffusions? Hence Table 1 may be misleading.\n- OT section is not clear and seems misleading:\n    - It appears there is no theoretical justification for being close to OT, am I right in thinking this is purely an empirical observation?\n    - Line 275 references Appendices C.4, however, there is no Appendix C.4 in the supplementary material\n    - Line 264, is this the Wasserstein 2 distance being compared? How is it computed? Hence is the ground cost the squared Euclidean distance? It is important to note that DDPM uses a discretized Ornstein Uhlenbeck process (VPSDE) which will result in an approximate OT distance with different ground cost to the squared Euclidean, one would need to use a Brownian motion (VESDE) for fair comparison, see e.g. section 3.1 of [1]. Hence the OT trajectory for different ground costs may not be a straight line, see e.g. [2], hence the diagrams and discussion may be misleading e.g. line 247. \n    - FBSDE, as advocated in [18]'s code, trains a forward and backward network to completion and then fine tunes with a small number of steps. If not training each network to completion, the final networks will not result in an SB approximation as it will not be following the IPF. In order to get a reasonable OT computation, one needs to train the network to completion at each IPF iteration.\n    - Perhaps comparing OT with closed form solutions such as between 2 Gaussian measures might be easier to justify, see e.g. Figure 2 of [1] (arxiv version)\n- Figure 4 uses a checkpoint from FBSDE [18], is this where the first iteration is using score-matching with a long time horizon and the forward diffusion in the first IPF iteration, then fine-tuned in subsequent IPF iterations? If so then visually looking at the diffusion time for non linearity may be misleading if most of the diffusion is close to Gaussian, see e.g. [2] the double well experiment for non-linear trajectories\n\n\n- Minor\n    - Line 179, \"Few works [18] in SBP detour the issue on the slow training by constructing an experience replay memory \", this technique for Diffusion SB was first introduced in [1], see page 53 Technique 3, and Algorithm 3. This paper was around 5 months before the reference cited.\n- Line 229, again [1] describes continuous time SB in section 3.5, and implements a discretized version\n\n\nSummary:\n\nThe paper proposes some nice ideas, and the method demonstrates excellent performance. The idea to use flows to convert a diffusion to latent is a good contribution.\n\nUnfortunately the authors focus too much on connections to OT (which appears theoretically ungrounded, not explained well, with some apparent misunderstandings) and not enough on explaining the actual method, algorithm, and training procedure, omitting key details from the main text such as the pretraining regime.\n\n\n[1] Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling\nValentin De Bortoli, James Thornton, Jeremy Heng, Arnaud Doucet, https://arxiv.org/abs/2106.01357\n[2] Solving Schrödinger Bridges via Maximum Likelihood, Vargas, https://arxiv.org/abs/2106.02081\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept