Maximum Likelihood Training of Parametrized Diffusion Model
Authors: Dongjun Kim
Byeonghu Na
Se Jung Kwon
Dongsoo Lee
Wanmo Kang
Il-chul Moon
Venue: ICLR 2022 Submitted (2022)
Keywords: Score-based Diffusion Model
Normalizing Flow Model
Variational Inference
Variational Gap
Stochastic Calculus

Abstract: Whereas the diverse variations of the diffusion model exist in image synthesis, the previous variations have not innovated the diffusing mechanism by maintaining the static linear diffusion. Meanwhile, it is intuitive that there would be more promising diffusion pattern adapted to the data distribution. This paper introduces such adaptive and nonlinear diffusion method for the score-based diffusion models. Unlike the static and linear VE-or-VP SDEs of the previous diffusion models, our parameterized diffusion model (PDM) learns the optimal diffusion process by combining the normalizing flow ahead of the diffusion process. Specifically, PDM utilizes the flow to non-linearly transform a data variable into a latent variable, and PDM applies the diffusion process to the transformed latent distribution with the linear diffusing mechanism. Subsequently, PDM enjoys the nonlinear and learned diffusion from the perspective of the data variable. This model structure is feasible because of the invertibility of the flow. We train PDM with the variational proxy of the log-likelihood, and we prove that the variational gap between the variational bound and the log-likelihood becomes tight when the normalizing flow becomes the optimal. 

Under review as a conference paper at ICLR 2022
MAXIMUM LIKELIHOOD TRAINING OF PARAMETRIZED DIFFUSION MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
Whereas diverse variations of diffusion models exist in image synthesis, the previ- ous variations have not innovated on a topic of their diffusing mechanism that was always kept linear. Meanwhile, it is intuitive that there would be more promising diffusion patterns to result in distribution closer to the approximating data dis- tribution. This paper introduces such adaptive and nonlinear diffusion method for score-based diffusion models. Unlike static and linear Variance Exploring (VE) Variance Preserving (VP) Stochastic Differential Equations (SDE) of the previous diffusion models, our Parametrized Diffusion Model (PDM) learns the optimal diffusion process by combining normalizing flow and diffusion process. Specif- ically, PDM utilizes the flow to nonlinearly transform data variable into a latent variable, and PDM applies the diffusion process to the transformed latent variable with the linear diffusing mechanism. Subsequently, PDM enjoys nonlinear and learned diffusion from the perspective of the data variable. We train PDM with a variational proxy of the log-likelihood, and we prove that variational gap between the variational bound and the log-likelihood becomes tight when the normalizing flow optimal.
INTRODUCTION
Diffusion models have recently achieved successes on task of sample generations, and some researches claim the state-of- state-of-the-an performance over Generative Adversarial Networks (GAN) (Karras et al. 2019). This success is highlighted particularly in the community of likelihood- based models, including normalizing flows (Greié al,, (02T), autoregressive models (Parmar et al. 2018), and variational autoencoders (VAE) (Vahdat and Kautz, 2020). Moreover, this success is noteworthy because it is achieved merely using linear diffusing mechanisms, such as VESDEs (Song and Ermon 2020) and VPSDEs (Ho et al. 2020).
This paper expands these linear diffusing mechanisms of VE/VPSDE to a data-adaptive trainable nonlinear diffusion. To motivate the expansion, though there structural similarities between diffusion models and VAEs, the forward diffusion in diffusion model has not been trained in existing literature while its counterpart, which is the encoder of VAE, is trainable. Rather, the current diffusion models assume the linear diffusing mechanism to be fixed throughout training procedure. Because of this static nature of the diffusing mechanism, variational gap (Cremer al. 2018) between the log-likelin log-likelihood and the Evidence Lower BOund (ELBO) remains to be strictly positive unless the score perfectly estimates the data score. This variational gap prevents the score training from being Maximum Likelihood Estimation (MLE). This gap is a fundamental motivation to develop a trainable encoder of the diffusion model, so that the gap can be tight.
As we tighten variational gap by training a nonlinear forward diffusion, the biggest challenge comes from an intractable optimization loss. The denoising diffusion loss requires a transition probability to be closed-fore in order to achieve fast optimization, but a nonlinear diffusing mechanism, in general, has no closed-form perturbation probability. Hence, our innovation becomes designing a diffusion model with tractable loss while the diffusion is learnable and flexible, which we achieve by merging previous diffusion models and the tractable variable transformation.
Our innovation concentrates the theoretic and the practical aspects of diffusion, which is tractable, learnable, and flexible. Theoretically, we observe that a nonlinear diffusing mechanism can be transformed to a linear diffusion under invertible transformation, and vice versa. In practice,

---

Under review as a conference paper at ICLR 2022
implement PDM by merging normalizing flow model and diffusion model, which constructs tractable, learnable, and flexible nonlinear diffusing mechanism. From this construction, we prove that variational gap can be further tightened by training the forward diffusion, nonlinearly. Also, demonstrate the state-of-the-ar performance in CelebA on Fréchet Inception Distance (FID) (Heusel al. 2017). We summarize our key contributions, below.
PDM expands the scope of forward diffusion process from linear and fixed dynamics to nonlinear and trainable dynamics.
PDM minimizes variational gap between the log- likelihoodd and ELBO, lower than the gap of linear diffusion models because PDM learns a data-adaptive nonlinear diffusion.
PDM explicitly proves in Theorem 2 that the optimal generative distribution equals the real distribution, which was not the case for a linear diffusion model.
2 PRELIMINARY
diffusion model is constructed with bidirectional stochastic processes. The forward direction diffuses an input variable to noise variable, and the reverse direction denoises the random noise to construct realistic input data instance. The diffusion model learns this reverse direction (generative process) by estimating data score. We provide brief summary of diffusion models, as below.
Forward Diffusion At the beginning of model build-up, diffusion process requires a diffusing mechanism on an imput variable. This paper assumes that the diffusion process is governed by a SDE of dx, f(x,, 1) di + G(x,, 1) dw, with Xg ~ Pr. where pr is the distribution from a real-world dataset; and x, is the solution of the SDE.
Reverse Diffusion The theory of stochastic calculus guarantees that we could create an identical diffusion process, (x-)r=o by solving an associated reverse SDE backwards in time (Anderson 1982). The associated reverse SDE is
dx; = [x(x t) G(x,, t)
(xx, (x,""(,, Vx, log (x:))] dt + G t) dwa XT ~ PT,
(1)
where Wi and We are standard Wiener processes with time flows forward and backward, respectively; and pt probability law of x,.
Generative Diffusion A diffusion model approximates the above SDE to eventually yield an estimation the data distribution of pr- In SDE previous literature setup that drift and diffusion terms, and G, are determined a-priori in the forward diffusion. However, data score, Vx, log Pt (xt), intractable to compute, we estimate this data score with score network of (x,,t) in order to mimic the reverse diffusion with our generative process. This score network approximates the reverse diffusion by plugging the estimated score in place of data score with the below generative process:
dx θ = [[(x] t) G(x, tt (x? ()so() (x?,t)) t)] dt + G(x, dw;, x π.
(2)
The generative process starts from a prior distribution (T), and it constructs time-continuous random variables x by solving a SDE backwards in time. The generated stochastic process denoted by {x 0 1T" Yz and we omit 0 in the superscript if no confusion arises. With this generative process, we define generative distribution, x Po. as the probability density of the generated random variable at time f = 0.
Score Estimation We train the score network by variational bound of the log-likelihood, given by
Ex, log
< <c((x)) x = ;2:0) - -xxy [log π (xT)]
= ..' (()) 0) dt Exy log (xT)).
(3)
Exo [x)) Vx, log (x/|x)) [1 up to a constant, where Pos (x, Xg) transition probability from X to x,- Here, À is a weighting function that deter- mines the level of contribution for each diffusion time the overall diffusion loss, L(xx) Y7 X; 0) (Song et al. (020).. Variational bound holds when the weighting function is the likelihood weighting (g) (Song al. 202T), where G is scalar-valued, g function.
where Le (x((
2

---

Under review as a conference paper at ICLR 2022
3 MOTIVATION OF NONLINEAR DIFFUSING MECHANISM
Though it has long been theoretically and empirically grounded to train the encoder part in VAE, such solid ground is not accomplished in diffusion models. This section analyzes structural similarities between a VAE model and a diffusion model, which brings the foundation of data-adaptive nonlinear diffusing mechanism.
3.1
VARIATIONAL GAP OF VAE
Given Negative ELBO (NELBO) of Negative Log-Likelihood (NLL) in VAE as
log po (x(
S { log po (x|2)] + DRL (go (z)x)ll((z)))
(4)
lo pe (x + D DL (z) (z)x2)))l (z/x)),
(5)
have pair of interpretations this NELBO. First, when focus Eq. 4 NELBO 1) aids data reconstruction by optimizing (x)x) log po x [z] and NELBO 2) regularizes the inference distribution of the encoder to prior distribution by x (g (z) x) p(z)) On the other hand, if we concentrate on Eq. [] NELBO bounds NLL by approximating an intractable decoder posterior of po(2 with tractable encoder posterior of y (2)).. A vanilla VAE (Kingma and Welling 2013) assumes this approximate posterior to be a Gaussian distribution with mean and diagonal covariance estimated by amortized inference: Tø (z) x) = [/z (x)x), o (x)I). By expanding the flexibility of this approximate posterior into a variational family of general distributions, for instance, Rezende and Mohamed (2015) resulted in tighter NELBO, which leads the optimization of VAE closer to MLE, and their choice of a generalizable model was normalizing flow.
To connect this NELBO process. Having that VAE attains a stochastic process of bivariate random variables, (x, 2), NELBO reformulated to KI. divergence between two joint distributions modeled in bidirectional ways:
diffusion loss, restate NELBO in the language of stochastic
D DKL (prloplpo) (Du(Ppllee + Ex)
-Bpoox) log (x.
log L (z/x)))
(x
(x(x)
po (z 2
(6)
log Pr (z))((Z))))
po (x) (z)xx)
=Dxi z) (x 2)),
The inequality 6] i.e. (p. S D DKi(go(x lpe (x, z)), well-known by itself in the field of information theory (Duchi 2016) by the name of data processing inequality. This restated bound interprets that VAE is essentially
bimodeling approach of a joint distribution on the bivariate stochastic process, {x, 2). On the forward direction (x - 2), latent variable is conditioned a data variable, and the joint distribution is modeled by 40 (x, 2) = pr(x))e (x)x) On the reverse direction - x). generative data variable is conditioned on the latent variable, and the joint distribution is modeled by (x, z) p( (zz)) (x iz). Under this equivalent framework, we present an analytic tool measure the closeness of NELBO and NLL (see 55 by
Gap( Pe) -NELBO- NLL
=DRI(gs(x, 2))l (RI(Prl1p))
-Ep,(x) D (gs (z) x/xx (z (z|x))]
and we denote this quantity as variational gap (Cremer et al. 2018). Though it is one of central tasks VAE to minimize this variational gap (Kingma et al. 2016). only limited works (Burda et al., 2015; Neal 2001) have estimated variational gap due to its computational burden. We introduce the tractable computation of variational gap in the next section on diffusion models.
3.2
VARIATIONAL GAP OF DIFFUSION MODELS
This section constructs analogy of diffusion model to VAE. To begin with, observe that the classical data processing inequality in VAE holds on stochastic process with bivariate random variables on Anite-dimensional Fuclidean space. On the other hand, the time horizon of diffusion
3

---

Under review as a conference paper at ICLR 2022
(x(x) Forward Flow on Data Space
p(zo) Forward Diffusion on Latent Space, p p(ZT)
Reverse Flow on Data Space
Reverse Diffusion on Latent Space
Schematic view of PDM. Normalizing flow transforms a data variable to a latent variable, and linear diffusion (VE/VPSDE) diffuses the latent variable to prior distribution. Data generation starts from the prior distribution, and the generation denoises randomly generated noise to latent variable, and the generation eventually transforms the latent variable into the data space.
Figure I:
model is continuous, so it contains am infinite number of random variables in its stochastic process as (x, Y2 Fortunately, the unite-dimensione classical data processing incquality on the bivariate random variables is directly extendable to a stochastic process with a continuous index. Formally, an iffnite-variate joint distribution is called path measure (Oksendal 2013), which is defined as probability law on a space of continuous functions induced by {x, {x Therefore, the path measure an alternative concept of joint distribution for stochastic process with a continuous index. The path measure for the forward diffusion in a diffusion model corresponds to joint distribution for the encoder part in VAE, and similarly, the path measure for a generative diffusion corresponds the decoder part in VAE. If şe is the forward (inference) path measure, and if vo is the generative path measure; then the data processing inequality becomes
DKL (Prllpe) S DKI(H))p))
The real distribution of pr is marginal distribution of μa. O, and the generative distribution of Pe becomes a marginal distribution of Ie at & = 0. Under this concept, the data processing inequality means that distributional divergence of D Dxxr(uulle (Hal) between two path measures lower bounded by a divergence of DKL (prllpo) between two sliced path measures at t = 0.
Previous works (Song al. 2021 Huang to diffusion loss,
2021 regard this variational bound being equivalent
D (RI(u))) ((x.) 9²; 9²:0) + (PTl(TT
where G(,,,t) = g(t) is scaler function t, and X(t) is square of the diffusion term g²)... Since the previous diffusion models optimize the diffusion loss, the KL divergence of Dx (p][ an equivalent target of optimization the diffusion loss. Subsequently, gap as a training performance to tighten NELBS BBO to NLL by
credit variational
Gap( vo) DxL (v) DKz(p,llpo)
This variational gap satisfies (see Appendix B.2)
Gap(μ, mD) (x.)) Dr.
so variational gap minimized if an inference posterior, (a( (((xx) (xo), estimates the intractable
generative posterior, vo((x;) Xn)) However, unlike VAE, the inference posterior is trainable, which restricts variational gap to be strictly positive in previous diffusion models. We note that diffusion model provides tractable NLL using the Instantaneous Change of Variable (Song al. 2020). and this tractable variational gap becomes useful in analyzing diffusion model quality.
4 PARAMETRIZED DIFFUSION MODEL.
4.1 No CLOSED-FORN TRANSITION PROBABILITY ON NONLINEAR DIFFUSION
As far as we know, all previously proposed SDEs, including VESDE and VPSDE, are categorized as linear SDEs because they have linear drift term of f(x,, t). This drift term determines the dynamics of the diffusion, so data-adaptive diffusion could be attained simply by parametrizing drift and diffusion terms. However, a transition probability with nonlinear coefficients for a SDE, Por (x₂ (x)), is intractable in general, and diffusion loss becomes intractable optimize. In fact, it is worth noting that a closed-form of the transition probability exists only on a certain type of SDEs, like VESDE and VPSDE.
4

---

Under review as a conference paper at ICLR 2022
Nonlincur (data noise)
Nonlincar Generative
datal
.b.8
X
X
x
dxff fa(xx () Ga(x) dw,,
Ra GG2 to 4i]
Gglx? fid
Invertible Path
Invertible Path
Zo
hg(Xo)
hg (x()
du, (1)) dt + g(4) thet
dalf
2 0ee (K0ooii" e)
o(() daan
Zp
Zo
20
[inear Inference Path (latent noisc]
R π
Lineur Gimerative Path Inoise lateni)
Figure PDM attains a ladder structure between a data space and a latent space by the invertibility of normalizing flow. A score network estimates data score following blue arrows, and model generates images following red arrows.
4.2 PARAMETRIZED FORWARD DIFFUSION WITH NORMALIZING FLOW
To construct tractable transition probability while keeping nonlinearity, observe that a nonlinear diffusion can be decomposed into a linear diffusion and a nonlinear invertible transformation, where the transformation provides the flexible nonlinearity on a diffusion process. Figure [ and 2 provide schematic view of our model. Normalizing flow transforms a data input to latent vanable, and diffusion model linearly permeates the latent variable to a noise distribution.
Concretely, let us define to be transformed latent variable, 2p = (xx)) where & is flow model parameters. Then, canonical diffusion with linear dynamics of
dz; 1 (t)z, dt gtt dw,,
Zo (Xo) with X Pr:
describes diffusing mechanism of latent variable. Since the transformation invertible, if we define an inverse random variable by x = (2.), the dynamics a stochastic process of {x, ) governed by a SDE of
dxf (x 1) de + Gx dwr.
x
Prr
where drift and diffusion terms are derived by
tx(,), = - [
Who
hos + (V²
(x()) = g [hv]]]]
from the Ito's lemma (Oksendal (2013). See full derivation and explanation in Appendix [B.] This diffusion process is induced diffusion on the data space that is defined indirectly by transforming back from the latent space to the data space. The nonlinearity of this induced diffusion fully depends on the invertible transformation as illustrated in Figure 2
It is not straightforward to directly construct generative reverse process on the data space, x Figures| [ and [ build the generative process by red arrows using the transformation. Starting from a prior distribution of denoise random variable from the prior to z according to the generative process of the linear diffusion on the latent space, and we transform the fully denoised latent of z back to the data space x h (z
4.3 TRACTABLE VARIATIONAL BOUND OF PARAMETRIZED DIFFUSION MODEL
The joint modeling of normalizing flow and diffusion enables a tractable training of nonlinear dynamics. The log-likelitee likielihoood on a data space reduces to the log- likelloodd on a latent space by adding the log determinant of Jacobian to the latent log- nog-likelihood. This calculation of liceliiis provides Theorem that bounds the log-likelihood with a tractable ELBO.
Theorem Suppose that Po. (xxo) is the log- bolliiklloood of generative random variable x Then, the log-tikee tigeiiklloood is bounded by
EE (x)) lo. Po.e (xx)) < ((xx) (bb
5

---

Under review as a conference paper at ICLR 2022
where
[((x))==: ::: (0, pxpr(
log det
Th Oxo
)
+ ((2x))/00
Exz
log ग (ZT)]
with ((2z)]=0) go::::::000 0) S (1)
[ [1(so(z,,))
Vz, log poz (2,
[20)] dt, up to a constant.
Here, Por (oo is a transition probability of the linear diffusion process in the latent space.
PDM ELBO in Eq. is different from the vanilla diffusion ELBO in Eq. from three perspectives. First, the normalizing flow loss, i.e. the log determinant Jacobian, is added to the loss of PDM ELBO. This normalizing flow loss represents transformation of data space into a latent space. Second, the remaining terms, related to the diffusion model, is a linear diffusion on the latent space with linear dynamies, which provides tractable optimization, rather than an intractable data diffusion of nonlinear dynamics. Third, the optimization loss in Eq. 7 estimates the score on the latent space by score network of Sg (Ze: (), while the diffusion loss in Eq. estimates the score the data space by score network of se (X,,t).
4.4 OPTIMIZATION LOSSES
Previous works (Ho ct al. 2020; Song ct al. 202. Vahdat et al. 2021 have empirically demonstrated
that training a diffusion model with a likelihood weighting. X(t) (t). could bring worse
performances than applying other weighting functions. For instance, Ho et al. (022)) Song and
Ermon (2020) introduce a variance weighting function, X(t) = o (t), and Ho et al. (2020) show that this weighting outperforms likelihood weighting in terms of the sample generation.
Within our framework, we also observe that a variance weighting function is advantageous over a likelihood weighting. Henceforth, we optimize flow model and diffusion model with different losses as follows:
Flow: min [(xx) o ( 0})
Diffusion:
min C( {g² or )²):0),
Note that the first term of the log Jacobian and the third term of the cross entropy in Eq. are independent to score parameters, O. Therefore, diffusion parameters are not influenced by these terms, which only the second term remains to update the diffusion parameters.
4.5 TIGHT VARIATIONAL GAP OF PARAMETRIZED FORWARD DIFFUSION
This section theoretically analyzes variational gap of PDM. To construct variational gap of PDM, we enumerate few nomenclatures for path measures (see Table Cp. On data space, 14 path measure for an inference process; and v3 is a path measure for generative process. On a latent space, 2 path measure for inference process; and v is path measure for a generative process. Here, v does not depend on & because the generative diffusion on the latent space starts from the d-independent noise distribution (T); and because the diffusion proceeds a denoising process by following a score estimation of sæ (Z,, t) that depends only on O.
Variational gap the data space becomes
(000) -NELBO- NLL Gup vd
(8)
-D (p (1olle - DKL PeLLPrllPo.o
(9)
where the subscript in Gap,, represents the data space. Recall that training the encoder in VAE guarantees tighter variational gap because the encoder distribution of gs z) approaches toward the decoder distribution of pe (x, z) by minimizing NELBO of Dx (xx(x, z z)lppe (x, However, this analogy is not valid in variational gap in Eq. 8 because both inference and generative path measures depend on Φ. This means that the generative path measure could arbitrarily deviate from the inference path measure when we optimize Φb

---

Under review as a conference paper at ICLR 2022
Using the change of variable, however, variational gap is equivalent
Capa(ug: v0,0) -Dxz(ulleg
(prlloop -D (H'sll))) D Dxi(polpe)
-Gap,( vD)-
With this derivation, optimizing an invertible transformation guarantees tighter variational gap because the generative measure of v is now fixed on Φ. Also, we note that this variational gap reduces to a KL divergence between the inference posterior and the generative posterior by
( v a) Gap v/)) = Ezo [DxL (H ((ze) |zo)) (lvl [2o)]]
(10)
which provides another analogy VAE. We enumerate variational gaps of the explained models in Table 5 In fact, we show in Theorem 2 that the optimum of PDM guarantees that the generative distribution pær.o o0 equals to the data distribution pr, and variational gap reduces to zero in the optimum. See Appendix A proofs.
Theorem 2. Suppose a flow network is flexible enough to transform to arbitrary continuous distribution. For any 0, (Vs((tt,)) is symmetric for any E [0,T] and V log (T)) so then there exists o² such that Gapa s va
wg .) = 0. Furthermore, we have
Φ- = arg min Gap v0,0) = arg min DEL PRILUllU - arg min DRI
To the best of our knowledge, Theorem is the first analysis in the community of diffusion models that the thcoretic optimum of a model completely recovers the data distribution. To proceed this argument, let consider a linear diffusion model in data space with no flow model. Then, marginal distribution of the forward process is pr at / = T, which differs from a prior distribution of In addition, suppose that the diffusion model reaches
the optimum point, i.e., its score network estimates the exact data score by Sø- (x,, t) = Vx, log Pr (xt). Then, a generative transition probability equals to the groundtruth (reverse) transition probability of pro (xo (x) Therefore, the generative distribution, Po (xo) f xx(x )rro (xr) dxy, naturally diverges from the real distribution, pr (Xp) [pp(xx (propr (x |xT) dxr, because of the mismatch of pr and x. This implies that the original linear diffusion model cannot estimate the real distribution, pr (xx). In contrast to the original diffusion model, applying normalizing flow to a diffusion model enables PDM to recover the real distribution by Theorem
2 See Figures and for further illustration.
RELATED WORK
Xa
Inferesee Pull [cats
Pull
X
Enooder
Decodet
Recently, a diffusion model has been jointly modeled in conjunction with other deep generative models. There two major previous works on this line of research. First, Vahdat et al. (2021) (LSGM) Figure 3 applies diffusion model to the latent space of VAE. Since VAE does not attain invertible structure, LSGM is not suggesting the expansion of diffusing mechanisms. In other words, L.GMM cannot provide diffused data learning the middle of the process because L.SGM does not induce the diffusion process on the data space due to the lack of VAE's invertibility. Consequently, the analyzing tool for LSGM should be variational gap of VAE, rather than that of the diffusion model.
Lew Difleie
Zy
Z0
Figure 3: LSGM.
Second, Song el al. (202T) (ScoreFlow) in Figure simply con- catenate normalizing flow on top of diffusion model provide a dequantized data distribution only at the initial diffusion time. De- spite of the structural similarity of ScoreFlow and PDM, major difference of ScoreFlow and PDM lies on their density model con- structions: Pg in Scoreflow and Po.e e in PDM. ScoreFlow constructs pe only with diffusion model, so the flow and the diffusion networks are not not jointly combined in the loss function of f (x) foo13. (u(x) log pe(x+uu) du dx. where u a dequantization variable (o(u/x)
Inriorence Paet Gommrniio Exoiso
(data nsiied
yr
Yo
Figure 4: ScoreFlow.
that lies in unit cube, [0, 1)d In other words, ScoreFlow adopts the flow outside of model density, so the flow is independent to the model density. In fact, ScoreFlow can be best understood as a dequantization technique (Ho et al. 2019), whose goal is estimating the optimal dequantizing region
7

---

Under review as a conference paper at ICLR 2022
Table Performance comparison on CIFAR-IO.
SDB
Name
Model
NLL
NELBO
Gap
FID
Table 2: Performance comparison on CelehA
VE
NCSN+
Difl Baseline
3.45
4.43
0.98
2.20
PDM (FID)
Diff & Flow
3.04
3.09
0.05
3.36
Model
NLL
NELBO
Gap
HID
DDPM++ (FII)
Dift Buseline
3.29
0.16
NCSN++ (VE)
2.39
3.96
1.57
3,95
PDM (FID)
Diff & Flow
3,04
3.09
0.05
6.79
PDM (VE. FID)
2.00
2.09
0.09
2.50
PDM (FID, dec)
Difl & Flow-v2
3.11
3.17
0.06
4.43
DDPM+ (VP)
1.79
2.19
3.03
DDP (NLL)
Diff Baseline
3.05
3.22
0.17
9.68
PDM (VP. FID)
2.04
2.10
0.06
2.23
PDM (NLL) PDM (NLL., dec)
Diff & Flow
2.98
2.99
Diff& & Flow-v2
2.96
2,99
0.03
7.23
UDM
1,93
2.78
VP
DDGM
2.92
DDPM+ (NLL. IS)
Dift Baseline
2.91
3.10
0.19
5,70
CR-NVAE
1.86
PDM (NLL., 1S)
Difl & Flow
2.99
3.01
0.02
8.94
NCP-VAE
5.25
PDM (NI.., dec, 1S)
Diff & Flow-v2
2.94
2.95
0.01
6.84
NonssFlow-74-10
1.99
LSGM- 109M (balanced)
Difl VAE
2,96
Styleformer
3.66
LSGM-450MM (halanced)
Diff & VAE
2.95
2.17
LSGM-480M (NLL)
Diff
2.87
6.89
LSGM-450M (FID)
Diff & VAE
3.43
2.10
on [0. 1)d to minimize dequantization gap (Hoogeboom et al. 2020). Therefore, variational gap of ScoreFlow remains to be strictly positive because training the flow does not optimize the forward diffusing mechanism. On the other hand, PDM constructs flow-embedded model density of Pb.o. and PDM optimizes the loss of f p. (x) log fo.1ya Px.0(x + u) du dx that puts normalizing flow inside of the model density. Although we train PDM with a uniform dequantization (Theis et al. 2015) by default, we could apply the variational dequantization PDM as well.
EXPERIMENTS
This section quantitatively and qualitatively analyzes suggested PDM on CIFAR-10 10 (Krizhessky et al. (2009) and CelebA (Liu et al. (215)) 64 x 64. We compute NLL for density estimations in bits per dimension scale by solving probability flow (Song et al., 2020) backwards time, and we compare baseline performances of (Song et al. (2020) 2021) computed on uniform dequantization. For sample generations, we use the Predictor-Correen
algorithm (Song et al.) 2020) for PDM on VESDE, and we use the numerical ODE sampler of the probability flow (Song et al. 2020) with the RK45 method on VPSDE. Throughout experimental results, we use NCSN+ (VE) and DDPM++
(VP) (Song et al. 2020) backbones of diffusion models, and a pair of ResNet- flow models
(Chen et al. 2019; Ma et al.) 2020) as backbones of flow models. We give a full description on models for our experiments in Appendix We name PDM (FID) and PDM (NLL) for experiments that use weighting functions as o² and g², respectively.
6.1 QUANTITATIVE RESULTS
We compare PDM to baselines in Table on VESDE and VPSDE as linear diffusing mechanisms. Although minor performance degradation the sample performance, Table [empiricall demon- strates reduced variational gap. In all experimental settings, variational gaps significantly drop less than 0. On the other hand, variational gaps of the baseline diffusions on VPSDE for the FID model and the NLL model are 0.16 and 0.19, respectively. We conclude that PDM effectively decreases variational gap by training a parametrized inference path measure.
Table 2 compares experiments on CelebA to the current state-of-the-art baseline models in various disciplines: the diffusion model of UDM (Kim et al.) (222]) and DDGM (Nachmani et al., (021). D. the VAE model of CR-NVAE (Sinha and Dieng, 2021), ). the flow model of Densellow (Greié et al. 2021) and the GAN model of Styleformer (Park and Kim 2021). We observe pair of implications. First. reduces
training a diffusing mechanism with PDM largely reduces variational gap. In particular, this inference gap from 1.57 to 0.09 on the NCSN+ backbone with VESDE and from 0.40 to 0.05 on the DDPM++ backbone with VPSDE. Also, we emphasize that NELBO of PDM largely outperforms NELBO of the baseline models. Second, our PDM reports the state-of-the-arr performance with 2.23 FID score in the CelebA sample generation. This opposite trend on FID (compared to the results in CIFAR- 10) indicates that a nonlinear diffusion is more effective on datasets of higher dimensions. In fact, this is quite intuitive result because as the data dimension increases, the family of feasible diffusion expands exponentially, and the data-optimal diffusing mechanism is more likely to deviate from linear diffusions.
8

---

Under review as a conference paper at ICLR 2022
Daseline lireen P0s (porlinoot)
Seedlnne Eirean PD14 jiunlrooa)
coss Rog)
HKM (nont
NELBO
Not
Training lieration
Diflaiion time
Trairing berstion
Test NELBO
(b) Score scale
Figure 6: Comparison of the linear diffusion model and PDM on deep network.
Figure 5: Comparison of the linear and nonlinear diffusing mecha- nisms on a shallow network.
Table 4: Ablation study on flow layers.
1. parmm)))
NLT
NEI
PDM
3,19
3.30
4(12MM)
ResFlow
5.72
PDM
3.19
3.24
0.05
8.2.5M)
ResFlow
4.48
PDM
3/44
3,09
0.06
(a) Samples from za
(b) Samples from X
32 (10.6))
ResFlow
3.72
Figure 7: Non cherry-picked samples from latent/data.
6.2
QUALITATIVE RESULTS
Test NELBO To figure out an effect of nonlinear diffusing mechanism on the training procedure, Figure 5 (a) shows test NELBO by training iterations on diffusion model with 35M parameters. Without any modification on hyperparameters,
train PDM with shallow ResFlow (Chen et al.) 2019) of 3M parameters, and Figure (a) shows that the training on PDM surprisingly stabilizes test NELBO. Figure (b) illustrates that such training stability can be attributed to nearly identical score scales. As described in (b), the score scale on a data space is unbounded as the diffusion time goes zero, and this extreme scale variation eventually leads training instability the baseline diffusion models. On the other hand, adopting flow transformation on data space mitigates such extreme scale variations that destabilize training.
Figure shows test NELBO on a diffusion model with 107.6M 6M number of parameters. To stabilize the diffusion model, we apply the A optimizer (Loshchilov and Hutter 2017) with 0.01 weight decay, and Exponential Moving Average (Song et al.) 2020) of 0.9999. Using the ResFlow model with 10. 6M parameters, Figure indicates that PDM consistently outperforms the baseline diffusion in terms of NELBO with smaller variations.
Flow Depth The flow model architecture is a building block that fundamentally determines the family of inference path measures. Table 4 Tempirically demonstrates that an inflexible variational family results in large variational gap on CIFAR-I0. Additionally, the absolute magnitude of NLL performance keeps decreasing the number of layers (L) increases.
Sample Generation Figure 7 presents samples from (a) a latent space and (b) a data space. Since a sample generation on the latent space is unnormalized, we normalize the sampled latent variables to be bounded on the range of [0, 255], so we can visualize the latent information in image. Figure indicates that diffusion model and flow model have distinctive and complementary roles in sample generation: 1) the diffusion model constructs the global context and 2) the flow model enriches this global context by colorizing into realistic image.
CONCLUSION
This paper expands the linear diffusing mechanism to be nonlinear by combining an invertible transformation and diffusion model. This nonlinear diffusing mechanism learns the forward diffusion process out of variational family of inference path measures, and such optimization provides tighter variational gap compared baseline model with linear diffusion.

---

Under review as a conference paper at ICLR 2022
8 ETHICS STATEMENT
As our work improves sample quality, our methodology could be a source of fake datasets, i.e. fake images. This is common threat from the line of deep generative modeling.
REPRODUCIBILITY STATEMENT
Every proofs and derivations explained in details Appendix A and Also, for reproducibility.
we would release our code as soon as the discussion period opens through an anonymous repository reviewers. We note that most of code built based the released code of Song al. (2020)
in order to compare with baselines in fair setting introduced in Song et al. (2020) and SSng et al.
(202I). Additionally, we provide the details on traning/vvaluationnnennr architecture in Appendix C
REFERENCES
Anderson, B. D. (1982). Reverse-time diffusion equation models. Applications, 12(3):113332.
Stochastic Processes and their
Burda, Y., Grosse, R., and Salakhutdinov, R. (2015). Importance weighted autoencoders. arXiv preprint met.i1509905599
Chen, R. T., Behrmann, J., Duvenaud, D., and Jacobsen, J.-H. (2019). Residual flows for invertible generative modeling. arXiv preprint vrXiv:1906.02735:
Cremer, C., Li, X., and Duvenaud, D. (2018). Inference suboptimality in variational autoencoders. In International Conference on Machine Learning, pages 1078-1086. PMLR.
Do Carmo, M. P. (2012). Differential forms and applications. Springer Science & Business Media.
Duchi, J. (2016). Lecture notes for statistics 31 /ellecticall engineering 377. URL: https://staffodd edudclass/stats3/ //Lectures/fulll notes. pdf. Last visited on, 2:23.
Grcić, M., Grubišíć, I., and Šegvić, S. (2021). Densely connected normalizing flows. arXiv preprint arXiv:2106.044 044:222066.06627. 04627.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.
Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, (2019). Flow++: Improving flow- generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pages 2722-2730. PMLR.
Ho, J., Jain, A., and Abbecl, P. (2020). Denoising diffusion probabilistic models. arXiv preprint arXhv:2006.1222
Hoogeboom, E., Cohen, T. S., and Tomezak, J. M. (2020). Learning discrete distributions by dequantization. arXiv preprint arXiv:2001. 11200001...1.........
Huang, C.-W., Lim, J. H., and Courville, A. (2021). A variational perspective on diffusion-based generative models and score matching. arXiv preprint arXiv:2106.02808.
loffe. S. and Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456.. PMLR.
Karras, T., Laine, S., and Aila, (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4401-4410.
Kim, D., Shin, S., Song, K., Kang, W., and Moon, L.-C. (2021). Score matching model for unbounded data score. arXiv preprint arXiv:2106.05527
10

---

Under review as a conference paper at ICLR 2022
Kingma, D. P. and Dhariwal, P. (2018). Glow: Generative flow with invertible 1x1 convolutions. arXiv preprint arXiv: 10X.:v8877.0333
Kingma, D. P. Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., and Welling, M. (2016). Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-4751.
Kingma, D. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., and Aila, (2019). Improved precision and recall metric for assessing generative models. arXiv preprint arXiv: /904.0691.
Liu, Z.,, Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738.
Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint UrXiv:1711.05:
Ma, X., Kong, X., Zhang, S., and Hovy, E. (2020). Decoupling global and local representations via invertible generative flows. arXiv preprint arXiv:2004.11820.
Nachmani, E., Roman, R. S., and Wolf, L. (2021). Non gaussian denoising diffusion models. arXiv preprint 1r..v:2006.0758..
Neal, M. (2001). Annealed importance sampling. Statistics and computing, 11(2):125-139)
Nichol, A. and Dhariwal, P. (2021). Improved denoising diffusion probabilistic models. arXiv preprint a7X.v.2102.09672.
Oksendal, (2013). Stochastic differential equations: an introduction with applications. Springer Science & Business Media.
Park, J. and Kim, Y. (2021). Styleformer: Transformer based generative adversarial networks with style vector. arXiv preprint arXiv:2106.07023
Parmar, G., Zhang, R., and Zhu, J.-Y. (2021)) On buggy resizing libraries and surprising subtleties in fid calculation. arXiv preprint arXiv:2 /222.
Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, L., Shazeer, N., Ku, A., and Tran, D. (2018). Image transformer. In International Conference on Machine Leurning, pages 4055 4064. PMLR.
Rezende, D. and Mohamed, S. (2015). Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR.
Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pages 234 141. Springer.
Rudin, W. et al. (1964). Principles of mathematical analysis, volume 3. McGraw-hill New York.
Sinha, S. and Dieng, A. B. (2021). Consistency regularization for variational auto-encoders. arXiv preprint arXiv:2105.148855
Song, Y., Durkan, C., Murray, I., and Ermon. S. (2021). Maximum likelihood training of score-based diffusion models. arXiv e-prints, pages arXiv-2101.
Song. Y. and Ermon, S. (2019). Generative modeling by estimating gradients of the data distribution. arXiv preprint arxiv: 190.1090
Song. Y. and Ermon. S. (2020). Improved techniques for training score-bare based generative models. arXiv preprint arXiv:2006.090/1
11

---

Under review as a conference paper at ICLR 2022
Song. Y., Sohl-Dickstein, J., Kingma, P., Kumar, A., Ermon, S., and Poole, B. (2020). Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201
Tancik, M., Srinivasan, P. P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ra- mamoorthi, R., Barron, J. T., and Ng. R. (2020). Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.1773 10739.
Theis, L., Oord, A. v. d., and Bethge, M. (2015). A note on the evaluation of generative models. arXiv preprint I
Vahdat, A. and Kautz, J. (2020). Nvac: A deep hicrarchical variational autoencoder. arXiv preprint 200iv:2007..38888.
Vahdat, A., Kreis, K., and Kautz, J. (2021). Score-based generative modeling in latent space. arXiv preprint orXiv:2106.05931.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and
Polosukhin, I. (2017). Attention is all you need. Advances neurul information processing systems, pages 5998-6008.
12

---

Under review as a conference paper at ICLR 2022
A PROOFS OF THEOREM AND THEOREM 2
Theorem 1. Suppose that Po (xx) is the log-likelihood of the generative random variable x-6" Then, the -iieliiheohd is bounded by
E (x)( log Po. I Ls ((x))((2==00
where
L0.0 (ox0((x)) (x)))=o: (x)) log det
Ohp Ox,!
||
+ Lo ((zz)t(oo))))) -Rzr [log x (ZT)
d {/ 3(t) (t)
+ 2
(2) dt,
with - + (1)(()) - B(x) ds.
(())
Proof of Theorem
7
From the change of variable, the transformation of Zu h induces
Po (zg)
(xxo))
det
Tha 27c0
and thus the entropy of the data distribution becomes
/ pr (xx) log pr (Xxo dxc
H(p,))
/ po (20) log
Po (Zu)
dzo
det
Uxg
f pr (xx)
dxoo - ff po po(z)) log po (zo) czzo
log
det
axx
|] / po(zo) log po (zo) dzo
(xo)
(xo)
log
det
0xx
Th
EE Ex, (Xo)
log
det
H(p))-
axn
From Theorem 4 of Song et al. (202T), the entropy at L = 0 equals to
H(po) H((p) 2 /'" (p.(..) 2Ve. f(z,,t) t) + g²( log (o dt,
where f(z,, t) is a drift term the diffusion for z, and pt is the probability distribution of z.. Therefore, the negative log-likelihood becomes
(xo) logg Po.0 po.o (xx)] (KILPp.lpo.o) H(p.)
S DKI vd %.0)) H(pr)
( vd (000) (x,( log det
Thp Oxo
|| + H((p)
D - Ep-(xo) log det
Ths
| + H((T)
ExQ
1 E d,3(t) + (t) (egp((2))1) dt.
2 2
Now, from Theorem 1 of SSng et al.
202T), the KL--divegeence between the path measures becomes
L (" g (t))))))))))) [Iso(ze, t) Vz, (o dt,
DRI(HollVb) (KI(PT(
13

---

Under review as a conference paper at ICLR 2022
so if we plug in this into the negative log- lilk-iikhhhd,,,, yield the following:
- (xo)) log loggs, P (xo)]
Th
{
T (() (2)) Vz. log dt
-IB..
log
det
Oxx
+DKI(pr(1z) H(PT)
Ez, 1 -d(311 (1)]V (z)))(
2 J0
-Ep. (xo)
log det
Th
)
-Ezr log ((g)(2T)]
+ d /'- (())
Oxx=
2
1
V logp: V log l. dt /'' (t))Ee [|s0 (z.,
+
2
Also, we have
Ezg so (zc, Vz, pog))(z:)] pe(z)sso(ae)) Va, lo dz
s0(2,;)) Vu.Pe(2s)
sa(ze, t) f po po((o) V Por (zo) dzo dz,
Ezg Ezs I20 (50 (2), Vn, log Por (2:120))
Therefore,
g² (t)lE. ||s0 1) 2z log):(Z:)l2 VV, pr
['" (() [ ||s0 (2+, 1)12 sa(2,, () Vx. log (zi)
(()) E22 (l)s((z::))
(zc. 1) so so(z,, t) Vz, Pog (Ze (20)
(-" T
g2 (e lao [|se(Z),)) 8) Vz, log Poc (z 20) V/ log por (z, (zo [12
Now, since Pue (z; mu)) = NNZz N()z: o² (t)II for v(t) and o² (t) determined by B(t) and g()), have
V1, log Poe (2: [20]] = Ez, |zu |||
2 μu( 2o
|| | (t o²(t)
d
E2 2o
= Ex(Z.
0² (t)
and we have the desired result.
Theorem 2. Suppose the flow network is flexible enough to trunsform pr to arbitrary continuous distribution. For any 0, if V so(;,)) ()) symmetric for any E [0, T] and V. log = (0 T), then there exists of such that Gapalu vgg
p o) 0. Furthermore, we have
Φ- arg min Gapalu vg.e) arg
arg
argmin min (KI (1 = arg min DKL (prlplllp.o)).
((tt) t) is a closed I-form. Therefore, from the Poincare's lemma (Do Carmo 2012). so t) exact, and there exists scalar field f
Proof of Theorem The symmetry of V (n(=tt indicates that (
that satisfies s. v.f((-) for any t. If define by of (z.) Txx (/_ (æ)) dz then have So (z, t) Vz, log of (zc, t) for any t. With this construction, the reverse SDE on the transformed latent space becomes
exp (/2 (2.))
dz =
- 3(1x2 --((() of log df (2 di + 9(1) dao,
[-
1
b = GT
0
14

---

Under review as a conference paper at ICLR 2022
Prolability in Space
(harwad)) ivfenenes Meauer Dosied Gexeratvee
Hypotiessi Space Generative Path Mensures ( Voly x]
Pr
By Pat
Pp"
Po
Updatec
Optimal Generarive Path
Forward Path
Time
Measure
Vir
(a) Optimal Generative Distribution Not Equal to Data Distribution
(b) Forward Path Measure Not Included in Hypothesis Class of Generative Process
Figure 8: (a) For the vanilla linear diffusion model in the data space, pr and æ naturally diverges. the score network estimates the exact data score (i.e., (x,,, () = V po (x,)), then the (optimal) generative distribution becomes pe (x)) = pr(x))pro (x)
(Xo) xr) dxy. On the other hand, the real distribution p-((x)) = ((x)) Xg Xxy) dxr, with identical (reverse) transition probability, pro (xol XT) to the generative process. Mismatch of pr and yields that the optimality of the diffusion model does not guarantee to estimate the real distribution: po(Xo) of л (xo). (b) Since pr and distinctive distribution, the forward path measure fu is not included in the hypothesis class of generative path measures, {Va the marginal distribution ve at is TH J.. addition, f is not the element of the closure of the hypothesis class, and conclude that D k lIpo) 0.
Spaceffhh Masruse Tadjulted"
(Reveac) Gencrative Pull Menare
Hopetiok Spor ofGon Misrato
(0())
(*7(0))
hyokdayri Space
8
Pg. Pr.
(b) Illustration of Theorem Existence &² any such that DI.. (pe po-11p4-.00 0
(a) Nonlinear Diffusion in Data and Latent Spaces
(a) While pr and is different in PDM as well, the marginal distribution of Pa f depends on the transformation, hg, modeled with normalizing flows. This d-dependent initial random variable at Zo influences its fully diffused variable of ZT. and optimizing á minimizes the discrepancy between Pr and т, where Zy Pr. Theorem 2 guarantees that this discrepancy is completely eradicated with the optimum point of Φ². (b) Theorem gaaremmmguarateee that for any 0, there exists of such that DrL.
Figure 9:
o) = O, which implies that p. (x() P .0 (xx)).
If pe is the path measure of the solution of the above SDE, then the marginal distribution of po at would be of Therefore, if hat maps pr to of then the forward process on the latent space
{z. Y becomes the generative process on the latent space {z ) Thus, we have 1 v/ in distribution, which leads D D ((/g) = 0. We get the desired result by the change of variable
formula: DKL (H (0g-o) = DKL ( (1 =
15

---

Under review as a conference paper at ICLR 2022
B DERIVATIONS
B.1 DERIVATION OF NONLINEAR DIFFUSION TERMS
Throughout this section, we omit Φ for the notational simplicity. Beginning with the linear diffusion on the latent space given by
1 ft)zzz dt + g(t) dws, Zo h(xo) with Xo Pr.
dz
(11)
2 have defined h (2+) the induced random variables. Then by the multivariate Ito's Lemma (Oksendal 2013), the k-th component of the induced variable satisfies
dx, Thg
dt +
V h (z)]
dz + tr (V h (zz) dz, dz?")
(12)
dt
2
Plugging the linear diffusion SDE in the SDE of Eq. Eq. 12 is derived by
{ (())z dt + g(t) dwo ) + g (()ur (V2, h4 (
dxr,a VV, h
(2x))² (z)))
(z:)) dt
1 --00 hi (z))"² 2, (()ttrr (V2, hy (2i)) di + a(1) [2, (z+)]² dws
(13)
abg because
Then, Eq. 13 in vector form becomes
0t
dx; f(x,, t) dt + G(x,, dw.,
where vector-valued drift and diffusion terms are given by
{ f(xi, t) = - - h (z) (z))z= + (t) (V2, (2:)) (((x))))=(((v.... (2.),
(14)
where V2, h=,)) is a 3-dimensional tensor with j. element to be (V hi (zx) i.j and the
T
trace operator applied on this tensor results in the vector of [r([2,, From the inverse function theorem (Rudin et al. 1964). the Jacobian of the inverse function V h (z,) equals the inverse Jacobian Vx, hxx)) Therefore, Eq. 14 is transformed
2: NV2,,TT' (z))))
tr ( h (z:))
{
f(x,,t) t) [v, h(x:) G(x,,t) = ( h(x))
h(xx)) + (t) (V h h (z.))
(15)
Now, we derive the second term of f in terms of x, as follows: observe that Σx Bac, 8x..3 where ōG, if 1 j and 0 otherwise. Differentiating both sides with respect to X4,1, we have
Oh, abg 8..j
Σ
Ōh,"
Ohe
Oh,"
The Oxm.
}
Oxi,
Oxx.5
dze,k
where the first term is
Thm
Oh, Ohg Oz,
(vvx)((x))) (V2, (ze)) m.k (xx)((x)) kgg
Ox,,1
tzc.
k.m
k,m
using the chain rule, and the second term becomes
Σ Oh.1 { (xx,,)
(
Ohe
)}
Σ -
(Ve.h- ((z)) v. (V2, ho()))
k Özzk
(xxe,s)
From above, we derive the trace term of in Eq. 15 as
2 h h ((x)) -tr (Vvx,
([xx)
V2 ((,)()))) Vx. -t),
16

---

Under review as a conference paper at ICLR 2022
Table 5: Comparison of variational gap between the VAE, the diffusion model, and PDM.
Models
Diffusion
Stochastic Process
Variational Gap (=NLL NELBO)
VAE
No
{x,z)
Ex
Diffusion Models
Data space
{xx){ {x (2)
Exu D (x))( [xo)]]
PDM
Latent space
Exo D (H ((2,) (20 ko))
where V2 x, h(x,) is 3-dimensional tensor with (i, element to be (V (x,nnxx()))))) inj Also,
define * operation as the element-wist matrix multiplication given by
h[x,)( V2 h(x:) 5. := Vx, [xxx)) ( [h(x))]
h(xx)) i.j
Combining all together, thus, we derive nonlinear drift term in Eq. 13 as a function of x, given by
1 t(x,,t) = - 2 3(1 h(x:)] h(x,)
1 (t) ([Vx, (VV, ([Vx,((x,]] h(x,))) [Vx, [x,,(x,]] -1)
B.2 DERIVATION OF VARIATIONAL GAP
First, the KL divergence between two path measures satisfies
/ (((xi)) log nu((x))) dxo:T vo((x)))
D (KI(u))
/ (x)) x0) log (x(x)) (( (xx H200
dxo:T
(x vol [xxe
/ (x) xx)
log (x()
log (1 Hxx)
dxo:T
px(
vo (7(()))) (xo)
/ (xo) log (x0) dxo + (xx))(((x))) (x0) log [x((x))) (x))) dxp:
po (xo)
( xx)
D kz (pr Ex,(x)) [De (Ml (x)xxx)) (x)
where No.T = fx )o Therefore, variational gap derived by
Table 6: List of path measures.
Space (Diffusion)
Gap(u, vo) -DKL (KL(H( -Dz(prllpe)
Mcasure
Type
-Ep, (xo) DKL (al (x( Xu)]| (o (xx))/x0))
Data (Nonlinear)
1
Inference
v
Generative
The gap Gapa(Hg v in Eq.
10 is derived to be
Latent (Linear)
Inference
Ez DK (Lz (Lze) (z0) analogously.
v/b
Generative
C IMPLEMENTATION DETAILS
C.I MODEL ARCHITECTURE
Diffusion Model We implement two diffusion models as backbone: NCSN++ (VE) (Song et al., 2020) and DDPM+ (VP) (Song et al.
2020 where two
backbones are one of the best performers in CIFAR- dataset. In our setting, NCSN+ assumes (gmaz 2 is the
the score network with parametrization of so 22, log 0² (t)), where (²()) = o² min amin
variance of the transition probability Pou (zz (2)) with VESDE. As introduced in Song et al. (2020), use the Gaussian Fourier embeddings (Tancik et al. 2020) to model the high frequency details across the temporal embedding. DDPM++ models the network with parametrization of (z,, 1). which
17

---

Under review as a conference paper at ICLR 2022
targets to estimate (t) pz.) We use the Transformer sinusoidal temporal embedding (Vaswami et al. 2017).
We use the U-Net (Ronneberger et al. 2013) architecture for the score networks both NCSN+ and DDPM++ based on (Ho et al. (020). We stack U-Net resblocks of up-and-down convolutions with skip connections that connect the ddenticalddimensionn by applying the global attention at the resolution of 16 x 16. We use four U-Net resblocks with four feature map resolutions (32 x 32 to 4 4). There are eight and four resblocks at each resolution on CIFAR-10 10 and CelebA, respectively.
layers. Also, we follow HHo et al. (2020)
Flow Model We implement two normalizing flow models backbone: the ResFlow (Chen et al. 2019) and the decoupled ResFlow (Ma et al. 2020). Although Ma et al. (2020) makes use of the
Glow (Kingma and Dhariwal 2018) for their flow model, we replace this glow architecture with the
ResFlow, mainly because the ResFlow transforms the variables from the identity function. We name the experiments on this decoupled ResFlow as PDM (FID, dec) and PDM (NLL,, dec). Otherwise, we train the model with ResFlow as default. For the ResFlow, we drop three components from the original paper: 1) the activation normalization Kingma and Dhariwal (2018), 2) the batch normalization loffe and Szegedy 2015), and 3) the fully connected layers.
On the deep ResFlow, we use resolutions of 32 x 32 and 16 x 16 on both CIFAR-10 10 and CelebA. Otherwise, we use single resolution of 32 32 on other experiments whose models have networks with layers less than 32. We apply the LipSwish activation (Chen et al. 2019) that provides the Lipschitz coefficients less than unity. For the multi-GPU training, we the Neumann log- determinant gradient estimator, instead of the memory efficient estimator (Chen et al. 2019).
C.2 EXPERIMENTAL DETAILS
Training All the models are trained with batch size of 128 and EMA rate of 0.999. We train each model for around 10 days on four V-100 GPUs with 128Gb GPU memory for all experiments. At the initial training phase, train the diffusion model parameters with fixed flow parameters initialized by the identity until the sample generation performance saturates. After this pretraining phase, we start updating both diffusion and flow models. In this learning phase, we apply the learning rate scheduling after the sample performance saturation. For the diffusion model, we drop the learning rate from 2 10 to 10 5. For the flow model, we drop the learning rate from 10-3 3 10 for VPSDE and 5 x 10-5 5 to 10-5 for VESDE. We emphasize this learning rate scheduling takes additional performance boost.
VESDE and VPSDE have different training details, VESDE assumes min 10-2 on all exper- iments. VESDE has maa 50 on CIFAR-10 and maa 90 on CelebA. On the other hand, VPSDE assumes ((t) = 3min + (8maz - Bmin) with 8min = 0.1 and 8maz 20. Both VESDE and VPSDE truncate the diffusion time on [T in order to stabilize the diffusion model, where < = 10-5 and 7 = 1. On sample generation, however, DDPM++/PDM with VPSDE draws samples by proceeding the generative diffusion process up to c = 10 which corresponds to the diffusion variance to be amin
10 On the sample generation, sampler Song et al. 2220) for NCSN++/PDM with VESDE and the Probability Flow (PF) ODE sampler (SongerSSnngegaaa al. (2020) for DDPM++/PDM. Throughout the experiments, we provide the identical diffusion model structures to compare the baseline model and the PDM model in a fair setting.
apply the Predictor-Correcte (PC)
Evaluation While we use the EMA rates of 0.999 for both diffusion and flow models for training, we use this EMA only for the diffusion model on the evaluation mode because the flow model without the EMA on the evaluation gives better samples. One peculiarity occurs in the experiment on the decoupled ResFlow. We empirically find that turming off the EMA for both diffusion model and flow model on the evaluation mode provides better performances of 3.11 in test NLL and 3.17 in test NELBO as reported in Table I on PDM (FID, dec). With the EMA applied to the diffusion model, we have 3.15 for test NLL and 3.22 for test NELBO, which shows slight degradation on the density estimation performance.
We compute the FID score hased on Song et al. (020) for CIFAR-10 with the modified Inception V1 network in order to compare PDM to the basel (Song et al. 2020: 2021) in a fair setting. On the
18

---

Under review as a conference paper at ICLR 2022
other hand, for the CelebA dataset, compute the clean-FID (Parmar et al. consistently antialiased results.
2021) that provides
C.3 VARIANCE REDUCTION
Flow Training When train the flow network with La.0 ( ((xe)=ooig") this NELBO contains the integration Lo {z+) {z))==:g)) However, previous works on the diffusion models (Nichol and Dhariwal 2021) show that the estimation variance is largely reduced with the importance sampling, which leads the performance variations (Song et al.
2021 Concretely, the importance
sampling chooses importance weight that proportional
(0)
(²())² and estimates the integration
by Lo (f g²) -o (t) ((z))o dt Σn 0² (t,) ((ze)T=o: f), where t, is sampled from the importance distribution.
Throughout the ex- periments, we select min 0.01, and, Oaar 50, 90 for CIFAR-10 and CelebA, respectively.
For VESDE, satisfies ((t) = 0 and g(t) = minn Taain
Omaz )
2log
(mmax)
amin
The transition probability becomes Poe (z) = (Z Zo. (t)). where (² _ g2() ds onin (maan ya( 1]. Song et al. (2020) approximates this variance to be o (t) = mnnn [maa )
Omen
that the variance proportional of (L). Therefore, the importance weight follows the uniform distribution, and the importance sampling is equivalent with choosing the uniform l.
On the other hand, VPSDE satisfies ((t) Siin + (Bmaz - Pmin) )t with g = / /t(). Then, the transition probability becomes Poe [ze (20) (())a, (t)T), where μ(() e (e)) and
o² (t) = 1 - e- Jo ((s) ds Thus, VPSDE has the importance weight of (²(() 1-e P6)55"
20
3(1)
The Monte-Carlo sample from this importance weight is the solution of the inverse Cumulative Distribution Function (CDF) of the importance distribution
= (u)
(16)
1 (²(
1 ) (
u= F(t
ds
(17)
2 J (s)
where u is uniform sample from [0, F(. F()) is the antiderivative of the importance weight given by F(t) log (1 0..5 (1mma)-mmm)) ( 48min) 0.5t (Bmaz -Biinn + t.8min and is the normalizing constant given by
g2) dt 02(1)
Z =
log (1 - c
-0.5t (Amas -amia) ( )1tmmi)) + 0.5t (Bmaz - Bmin ) + t.3
= llgg (1 -0.57 9men) ) TTmmn log (1 (1- -e U.3e? U..ee"(Bmos (Bmon
Bmin) -e
+ 0.5(T - )Bmax 3min) (T - Bminn
233.86
for T = and = 10-5, The solution for the inverse CDF in Eq. 16 becomes
el ((5) ds
+ exp (Zu F(e))
S B(s) ds -5 Prinn Bmint log (1 + exp (Zu F(e)))
-min + / + 22(3maz - Bmin) log (1 exp (Zu + F(e)))
Bmaz
Bmin
The variation of the Monte-Carke diffusion time depends the uniform sample of u.
Diffusion Training Now, on the side of training the diffusion network, as described the main paper, selecting the weighting function heavily influences the model performances. For instance,
19

---

Under review as a conference paper at ICLR 2022
Table 7: Additional experimental results on the pretrained DDPM++ with VPSDE. The reported
+DPM++ result (Song et al. 2021 is not fully reproducible in density estimation performances.
NLL
NELBO
Gap
DDPM+ (NLL., 18, reported)
10-5
2.91
3.10
0.19
5.70
10-5
3.03
DDPM+ (NLL, 18, ours)
10-1
3.00
3.20
0.20
37.28
10-5
2.94
0.01
6.84
PDM (NLL, dec. IS)
2.95
2.86
0.10
8.14
Table 8: Additional experimental results for precision and recall.
SDE
Model
NLL
NEI
Precision
Recall
FID
NCSN++
0.67
2.20
VE
PDM (FID)
3,04
3.09
0.67
0.60
3.36
DDPM++ (FID) PDM (FID, doc)
3.13
3.29
0.68
0.60
3.16
3.11
3.17
0.63
0.61
4.43
VP
DDP (NLL, 18, ouns)
3.03
3.20
0.59
0.62
5.67
PDM (NLI., dec, IS)
2.94
2.95
0.59
0.61
6.84
Song et al. (2021) compare the weighting functions of g2 and o², and Song et al. (2021) conclude that the weighting function of X(t) = d²(t) works the best for the sample generation, and A(t) = gt( yields the best NLL performance. We train the diffusion model with o² (t) for PDM (VE/VP, FID), and with (t) for PDM (VP, NLL.) models.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 DETAILED COMPARISON AND ABLATION ON TOLERANCE
Table has two implications. First, we find that the re- ported NLL and NELBO performances slightly better than what we actually obtain in our pretraining step, de- noted by DDPM++ (NLL, IS, ours). Comparing with our results, optimizing PDM is indeed beneficial on density es- timation, and variational gap is significantly reduced from 0.17 to 0.01. Second, we find the tolerance of 10-1 for the numerical ODE solver on computing NLL is the sweet spot on obtaining best NLL of 2.86, which beats LSGM-480M (NLL) on density estimation. This noteworthy because our model uses 120M number of parameters, a quarter of parameters LSGM. On the other hand, NELBO is independent of the tolerance because it does not use the numerical ODE solver, and Table shows that variational gap is minimized when the tolerance is small enough.
NFE (datted line)
PCM (FIC) PCM (FII, dect
FID (solid lime)
Tolerance of RK45 ODE Solver
Figure 10: Ablation study on tolerance level of the RK45 ODE solver.
Figure 10 illustrates the ablation study on the tolerance for the ODE solver. Surprisingly, PDM is robust the tolerance level up to 10-1, in contrast to LSGM with poor sample quality at 10-1 (Vahdat et al. 2021). From this robustness on the tolerance, we could reduce the sampling wall clock time from 360s (tolerance 10-5) to 72s (tolerance 10-l) for 1,024 samples, which is 5x faster, with minor sacrifice in FID score from 6.79 to 8.56 in PDM (FID) with VPSDE.
D.2 PRECISION AND RECALL
While FID measures the overall sample quality that counts both fidelity and diversity at once, we provide precision and recall (Kynkäänniemi et al. 2019) based on the modified Inception VI network
20

---

Under review as a conference paper at ICLR 2022
Table 10: FID-IK 1k of PDM with VPSDE CIFAR-
Table FID-Sk PDM (VE, FID) on CIFAR-10.
SNR
FID-Ik
Temperature
0.14
0.145
0.15
0.155
PDM (FID)
PDM (FID, dec)
0.5
16.59
15.99
15.95
15.7
0.5
40.09
36,84
0.6
12.61
12.13
12.17
11.99
0.6
38.28
35.31
0.7
10.41
9.95
10.08
9.94
0.7
37.10
34.36
0.8
9.2
8.75
8.94
8.84
0.8
36,28
33.77
0.9
8.58
8.13
8.35
8.28
0.9
35.87
33,44
Temperature
1.0
8.31
7.85
8.07
8.06
1.0
35,57
33.24
1.1
8.27
7.84
8.01
8.07
1.1
35.40
33,21
1.2
8.49
8.11
8.21
8.31
1.2
35.28
33.24
1.3
8.97
8.61
8.67
8.84
1.3
35,28
33.24
1.4
9.76
9.44
9.45
9.73
1.4
35.38
33,42
1.5
10.93
10.68
10.63
11.01
1.5
35.68
33.65
Table 11: FID-1k of PDM (VE, FID) CelebA
Table 12: FID-Ik of PDM
(VP, FID) on CelebA.
SNR
Temperature
FID-Ik
0.16
0.17
0.18
0.8
15.60
0.8
17.37
16.88
17.8
0.9
15.27
0.9
15.37
15.04
16.02
1.0
15.04
14.72
14.56
15.37
1.05
14.95
Temperature
1.05
14.72
14.53
15.24
14.90
1.1
15.00
14.83
15.47
1.15
14.94
1.15
15.52
15.36
15.94
1.2
15.05
16.3
16.14
16.71
1.3
15.44
1.3
19.09
18.81
19.40
with 50k samples in order to separate the sample fidelity and diversity. Precision is metric that measures the sample fidelity, and the recall is for the sample diversity. Table indicates that both precision and recall remains the identical scale, except for precision at PDM (FID, dec).
D.3 EFFECT OF SIGNAL-TO-NONE AND TEMPERATURE
As the flow colorizes the latent sample, we experiment the effect temperature (Kingma and Dhariwal. 2018) on this colorization Figure II With temperature the normalizing flow puts its latent input scaled by the flow network. In PDM, the image color with higher temperature tends to be brighter, and we find that the optimal temperature depends on the experimental settings.
bus
Tempertture
3.0
On sample generation, we draw latent samples from either the PC sampling or the ODE sampling. sampling assumes the Signal- to-Noise Ratio (SNR) that modifies the Corrector algorithm (Song al. 2020) of the annealed Langevin dynamics (Song and Ermon 2019). From the sampled latent variable, we scale by multiplying the temperature (Kingma and Dhariwal 2018) to the latent variable. Since the diffusion model learns the global image structure. the normalizing flow only paints the color of the image, and the temperature controls the brightness of the generated image. This a contrastive result to the previous flow models (Kingma and Dhariwal 2018, Chen al. 2019) that the temperature in the previous models controls both global context (such as the smoothness) and color. Therefore, we can create images with different colors by manipulating this temperature.
Figure 11: Ablation study for the flow temperature.
Sinne already used this inception network in calculating the FID score, we compute precision and recall based this inception network. Note that we used this version of inception network to comply the haseline (Song al. 2020
21

---

Under review as a conference paper at ICLR 2022
Figure 12: Non cherry-picked random samples from CIFAR-10 trained on PDM (VE, FID).
CIFAR-10 With these hyperparameters of the SNR and the temperatures, find optimal values in the below Tables. Taalee9ccmmppl the FID-5k score of all combinations for (SNR, r) in PDM (VE, FID) on CIFAR-IO. Unlike the SNR of 0.16 that previously suggested in (Song al., 2020) for the vanilla diffusion model, we find that SNR of 145 consistently outperforming other values for SNR terms of FID. Similarly, Table Presentss that the optimal temperature is 1.1.
For VPSDE, Table 10 shows that the temperature of 1.3 performs the best for sample generation in PDM (VP, FID). For PDM (VP, FID, dec), we find that the temperature of 1.1 performs the best, and for PDM (VP, NLI., dec), the vanilla temperature of 1.0 generates the best samples.
CelebA Table TT shows the FID-16 performance of PDM (VE, FID) on CelebA. It shows that the SNR of 0.17 proposed in (Song et al. 2020) performs the best also in our PDM. In addition, it shows that the optimal temperature is either 1.0 or 1.05. Table 12 shows the FID- performance of PDM (VP) on CelebA. It is clear in this table that the sample quality is the best on the temperature range of 1.05 to 1.15.
D.4 RANDOM SAMPLES
Figure 12 and 13 show the non cherry-picked random samples from PDM (VE, FID) CIFAR-10 and PDM (VP, FID) on CelebA, respectively.
22

---

Under review as a conference paper at ICLR 2022
Figure 13: Non cherry- piccey random samples from CelebA trained on PDM (VP, FID).
23


Reviews:
[{"review_id": "jhfubX0KPI-", "reviewer": "Reviewer_LLCZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of injecting nonlinearity to a linear diffusion using normalizing flow is very interesting, although somewhat similar to [Song et al. 2021]. I find the experiments section quite convincing. In particular, the variational gaps obtained using the proposed method are a lot lower than the alternatives.\n\nI find the exposition of the paper very hard to follow. Many sentences do not make sense to me, and notations are used without being defined properly.\n\nDetailed comments:\n- In Eq (3), some reference about how $\\mathcal{L}$ appears would be helpful. Also, mention again how $\\mathcal{L}$ is defined in Sec 3.2 where it is used. Throughout, the role of $\\mathcal{L}$ is really mysterious to me.\n- Above Eq (6), \"To connect this VAE ELBO to the diffusion loss, ...\" How is (6) related to diffusion? I found the connection a bit far-fetched, if it's just regarding $q(x|z)$ and $p(z|x)$ are \"forward\" and \"backward\" directions respectively.\n- In Sec 3.2, how does the last equation follow?\n- In the last paragraph of Sec 3.2, I don't understand how the variational gap has to be strictly positive for diffusion models. Why couldn't $\\nu_\\theta$ be arbitrarily close to $\\mu$ as we optimize $\\theta$? How is this really different from the nonlinear case Eq(10)?\n- In the last sentence in Sec 4.2, \"from the prior to $z_t^\\theta$\" I think it should be $z_0^\\theta$.\n- Before Theorem 2, it claims \"this variational gap converges to zero as we train ...\". This does not seem correct. The theorem mentions nothing about convergence, but only that at optimal the gap becomes zero. The actual convergence is hard to guarantee due to the non-convexity of the optimization.\n\n\n", "summary_of_the_paper": "The paper proposes parameterized diffusion model (PDM) that combines a normalizing flow on top of a linear diffusion model, effectively modeling a nonlinear diffusion. Empirically, such nonlinearity results in a tighter variational gap compared to the linear diffusion and other baselines.", "summary_of_the_review": "The paper proposes an interesting way to model nonlinear diffusion using normalizing flows, and presents convincing experiments results to support the main claims, although the clarity of the writing can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gJ1h3Z4o8B", "reviewer": "Reviewer_qGLk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\nThe paper is clearly written and easy to understand. The overall idea and relationship to VAE are also discussed at the very beginning, which helps the understanding of the proposed method. For theoretical correctness, I have briefly checked the derivation, which seems to be correct. However, I am still a bit confused about some of the notations, which I will elaborate on later. Empirically, the author compared the proposed method with various diffusion models and demonstrated better NLL/ smaller variational gap and stable training. \n\n## Weakness and potential improvement\nI found the intuition behind the proposed method interesting and easy to follow, but I still have some concerns. First, if I understand correctly, the structure of PDM is very similar to LSGM mentioned in the related work. The only difference is the replacement of the encoder and decoder with normalizing flows. What are the actual empirical advantages? From the experiment, it seems that LSGM produces lower FID scores compared to PDM. Does PDM reduce the required number of generation steps like LSGM compared to typical linear diffusions?\n\nFor the evaluation metric, FID score is a good metric for overall sample quality, which captures both diversity and fidelity. However, since the PDM produces better NLL, I guess it can produce better distribution coverage than typical linear diffusions. So, if it does not trouble too much, **Precision** and **Recall** [1] are good metrics to separate the sample fidelity and diversity, which can provide more information than just FID.\n\nFrom the generated images before and after normalizing flow, it seems that the normalizing flow only learns to colourize the image. Thus, most of the image details are generated by linear diffusions in latent space. My concern is that is it possible that the normalizing flow only learns trivial mappings? Why does normalizing flow not contribute to image details? \n\nWhat is $\\lambda$ in Eq.3?\n\nAt the beginning of section 4.2, I don't think all non-linear diffusions can be decoupled into normalizing flow + linear diffusion. Thus,  it is only fair to say that normalizing flow + linear diffusion produce non-linear diffusion. \n\nWhat is $\\sigma^2(t)$ in section 4.4?\n\nIn section 4.5 on page 7, I am still confused about why optimizing $\\phi$ reduces the variational gap. I understand that the gap can be expressed as the difference in KL w.r.t. latent space path measures, but the variational gap also has the term $D_{KL}(p_\\phi||p_\\theta)$ which is also related to $\\phi$. So why optimizing $\\phi$ w.r.t. NELBO alone can reduce this gap?\n\n[1] Tuomas Kynkaanniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models.  \n ", "summary_of_the_paper": "The author proposed a framework that combines normalizing flow with diffusion models, allowing non-linear diffusions to be used for inference and generation. Specifically, the author first discussed the similarity between VAE and diffusion models to motivate the importance of using non-linear diffusion and the reducing variational gap. Then, the author proposed to decouple the non-linear diffusion into an invertible transformation with the normalizing flow and typical linear diffusions on the transformed variables. A training objective based on variational lower bound was also proposed. Further, the author also showed that the variational gap between the lower bound and log-likelihood can be reduced with the proposed non-linear diffusions. ", "summary_of_the_review": "The paper is clearly written and I am also interested in the actual advantages of the proposed method compared to other diffusion models. If the author successfully addresses the above concerns, I can improve my rating of the paper. \n\n---\nI have read the revised version of the paper and the author's responses. It addressed most of my concerns. However, although I understand the difference between PDM and LSGM from a theoretical point of view, I think demonstrating it empirically is also important because the inter-changeability between latent space and data space during inference is mainly due to the merit of normalizing flow, not the PDM framework. Since concatenating the flow and diffusion model is straightforward, the resulting model (from a structure point of view) has limited novelty. On the other hand, I think the argument of reducing the variational gap is interesting, and along with extensive empirical evaluations, I will raise my score to 6. But I still suggest the author to further argue about the novelty of PDM from both theoretical and empirical points of view. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dhkpvdlg9Kq", "reviewer": "Reviewer_4sPP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n- Combining a Normalizing Flow with a Diffusion Model as proposed in the paper has not been done before. \n- The model allows to calculate exact data log-likelihoods using the diffusion model's probability flow ODE together with the tractable log-likelihood of the Normalizing Flow component.\n- The deterministic transformation defined by the Normalizing Flow component allows to formally relate the combined flow and diffusion model to a non-linear diffusion directly in data space via Ito's Lemma.\n- It is interesting that the method results in small variational gaps between log-likelihoods and evidence lower bounds used for training.\n\n**Weaknesses**:\n- It is mathematically elegant that we can formally relate the method to a non-linear diffusion in data space via Ito's Lemma. But what is the practical value of that? Both training and sampling still decompose the model into its separate flow and diffusion components and do not leverage this connection. Also, the experimental results suggest that the model only reduces variational gaps, but does not improve the actual likelihood itself (for example, *DDPM++ (IS)* vs. *PDM (NLL, dec)* in Table 3, also see points below).\n- The method is very incremental compared to LSGM [1]. It is essentially equivalent with the only difference being that LSGM uses a VAE framework with separate, non-invertible encoder and decoder, while PDM uses an invertible neural network, so encoder and decoder are simply the inverse of each other. However, this has multiple downsides: (a) Invertible neural networks have significantly lower expressivity than free neural networks due to the invertibility constraint. Hence, PDM has less modeling power than LSGM. (b) Using free neural networks for encoder and decoder, LSGM allows to use smaller or larger latent spaces, which can affect sampling speed, and it also allows to tailor the encoder and decoder better to different data types. No such things are possible with Normalizing Flows. Compared to LSGM, the only advantage of the invertible and deterministic encoder/decoder networks (deterministic in contrast to VAEs, where encoder and decoder formally define distributions) is that we can use Ito's Lemma to define a non-linear diffusion in data space. However, as mentioned, this advantage is not practically leveraged in any way, it seems, and the model is still trained separating the regular flow and the regular linear latent-space diffusion.\n- Related to that, I think the statement \"LSGM cannot provide the diffused data learning at the middle of the process because LSGM does not induce the diffusion process on the data space due to the lack of VAE’s invertibility\" is a bit misleading. While it is true that the encoder/decoder are not formally exact inverses of each other, in practice they do almost perfectly invert each other as indicated by the high reconstruction quality of modern VAEs and behave almost like deterministic functions. Therefore, also a model like LSGM can easily take the latent variable from anywhere along the diffusion process and map it back to data space with the decoder, just like PDM. In fact, such an experiment is demonstrated in Figure 15 in [1] (therefore, also the strict comparison between the pipelines in Figure 2 and 3 is a bit questionable).\n- I think the paper misses some important baselines on the CIFAR-10 experiments: Recently, [2] achieved state-of-the-art log-likelihoods using diffusion models. This work should be cited and would also be an appropriate baseline. Furthermore, when comparing to [1], the authors choose only one setup that was tailored to very high FID at the cost of likelihood. However, PDM not only focuses on FID, but primarily at NLL, lower bounds, and smaller variational gaps. Hence, it would be more appropriate to also compare to *LSGM (NLL)* and *LSGM (balanced)* (Table 2 in [1]).\n- The experimental results are not overly strong. Considering these further baselines, the strongest competitive diffusion-based works [1,2,3] all achieve likelihoods or lower bounds around 2.90 or lower, while PDM's best value is 3.11. Also the FID is significantly higher than recent works in the literature. I don't think that only demonstrating small variational gaps is a significant result, as long as the actual performance is not competitive. Note that I am giving more weight to the CIFAR-10 experiments, since this is a widely used benchmark within the relevant literature (i.e. diffusion models), while CelebA 64x64 is not that widely used. \n\n**Questions**:\n- The NLL for *PDM (FID, dec)* is actually better than the NLL of *PDM (NLL, dec)* in Table 3. Why is that? Wouldn't we expect to achieve better likelihoods when optimizing with the \"likelihood\" weighting $\\lambda(t)=g^2(t)$?\n- Why does *PDM (NLL, dec)* achieve so much worse NLL/NELBO than *DDPM++ (IS)*? My understanding is that PDM uses a similar DDPM++ backbone, which is trained first (after this training stage, it should have similar NLL/NELBO as the pure diffusion *DDPM++ (IS)* baseline). Now, we additionally train the flow. Shouldn't it only further improve the results?\n- Based on Figure 7 and as discussed in the text, it seems the flow component's main job is to adjust colour saturation of the synthesized images. I am wondering whether this is related to the training strategy: In the appendix, it is mentioned that first the diffusion component is trained and only then the flow is added. Hence, the flow will afterwards only improve or \"clean up\" whatever the diffusion model missed and this seems to be related to colour saturation. What if we trained the other way around and first trained the flow to transform the data distribution into a Normal distribution (standard flow training), and then trained the diffusion model on the actually achieved embedding distribution? We may see different results here. Also, what would happen if we do not do stage-wise training but directly trained all components end-to-end simultaneously from scratch?\n- I am not sure I am fully understanding the value of Theorem 2: If we assume that our score model, i.e. diffusion model, is flexible enough to cover any path measure, then why do we require a flow component in the first place? With this assumption, the diffusion model should be able to model the data distribution itself (diffused for all $t$) perfectly without requiring an additional flow component, I would think?\n\n**Minor Comments**:\n- Equation 4 seems to be the negative ELBO, not ELBO.\n- In Section 3.2, the paper discusses the variational gap and refers to previous works on diffusion models. Previous work showed that the generative distribution defined by the probability flow ODE and the stochastic SDE formulation are not identical, unless the model has learnt the ground truth score perfectly, which is not the case in practice (see [3]). Note that the base for the KL derivations is the SDE formulation, while the NLL calculation in practice relies on the probability flow ODE formulation. I think we are essentially assuming here that ODE and SDE-based models are similar. I believe this subtlety should be highlighted.\n- Page 8, last sentence: Did the authors mean \"destabilize\", rather than \"stabilize\"?\n- Section 9: \"prodive\" -> \"provide\".\n\n[1] Vahdat et al., \"Score-based Generative Modeling in Latent Space\", 2021.\n\n[2] Kingma et al., \"Variational Diffusion Models\", 2021.\n\n[3] Song et al., \"Maximum likelihood training of score-based diffusion models\", 2021.", "summary_of_the_paper": "The paper proposes to combine Normalizing Flows with generative Diffusion Models in such a way that the target data is first non-linearly transformed via the flow and then the distribution over latent embeddings is modeled with the diffusion model. The authors call their model *Parametrized Diffusion Model (PDM)*. Since the flow is invertible and defined by a deterministic function, it is possible to formally relate the combined flow and diffusion process to a non-linear diffusion directly in data space, leveraging Ito's Lemma. The paper validates the PDM on image modeling benchmarks (CIFAR-10 and CelebA 64x64). The experiments show that the method leads to reduced variational gaps between the data likelihood and the evidence lower bound that is used for training the model, compared to several baselines.", "summary_of_the_review": "It is interesting that the flow transformation allows for a formal definition of a non-linear diffusion in data space via Ito's Lemma, but it seems this cannot be leveraged for improved modeling power, as directly training this non-linear diffusion is not possible, unfortunately. Hence, the paper resorts to training a regular flow together with a regular \"linear\" diffusion, separating the two components. With this in mind, I think methodologically the paper is fairly incremental compared to LSGM [1]. It is basically the same with the encoder/decoder replaced with invertible neural networks and the invertibility does not provide any practical advantages. The paper's experimental results are not impressive and I think some relevant baselines are missing. Therefore, in conclusion I recommend rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D6Asjzsvslx", "reviewer": "Reviewer_wDxP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is set up in the emerging toping of generative diffusion models, that is gaining some important momentum. Such models define a forward and a backward pass, that respectively consist in going from the signal domain to noise and back from noise to the signal domain.\nApplications of the backward diffusion as a generative model is a very effective and popular way to achieve generative modeling.\n\nAs highlighted by the authors, current applications of such diffusion models come with linear dynamics, since those were the only ones so far to allow for close-form expressions for the required densities as a function of the time step. Their point is that such linear dynamics are limiting the expressive power of such models.\n\nAs a convenient way to circumvent this limitation, their solution is fundamentally quite simple: they propose to jointly train a normalizing flow from the data space, from which a classical linear diffusion is performed. The key point is that the method actually turns out to be equivalent to a nonlinear diffusion that would take place in the data domain, and they explicitly derive its dynamics by exploiting the Ito lemma and the fact that normalizing flows are invertible. The paper comes with much theoretical derivations that should be sufficient to support their claims.\n\nAs can be seen, I tend to think that this paper does have merits and that the study definitely may be inspiring to colleagues working on diffusion models. Although the experiments are limited, they are sufficiently convincing to me.\n\nHowever, I must also say that all these remarkable developments come with an unacceptably low quality of english usage. A very important number of sentence from this paper comes with typos or with awkward usage of the wrong english words. \n* The most annoying systematic typo is the systematically wrong usage of the word \"the\". That word is very often written at incorrect places, which reads very awkward. I am  talking of something like a hundred of occurrences.\n* Then, there are other more punctual typos or awkward sentences, that I shortly review later below.\n\nFor the reason that this english usage is not acceptable I am sorry to say that I must recommend rejection. I however strongly encourage the authors to have the paper (heavily) proofread by a native english speaker and then submit it again somewhere else.  \n\nComments on the go:\n\n* \"by maintaining the static linear diffusion\": have not innovated on the topic of the diffusion mechanism that was always kept linear.\n* You don't define your acronyms VE VP, SDE, MLE, etc\n* Practically : in practice\n* In equation 3, you should explain what P_{0t}(xt|x0) is and how you compute it\n* more close to: closer to\n* To be concrete: in practice\n* All the suggested SDEs: As far as we know, all previously proposed\n*  worth to note: worth noting\n* in equation (7), how do you write and compute P_{0t}(z_t|z_0)\n* sentences before 4.4 awkward.\n* your ethics statements are largely exagerated. It reads like you are proposing a super dangerous technology, whereas you are basically proposing some variation over a quite established technology. Please strongly tune down. (\"destroy the evidence-based justice system\", come on...)\n\n", "summary_of_the_paper": "The core contribution of the paper is to show how a nonlinear diffusion model may be constructed by cascading a normalizing flow with a linear diffusion in the resulting latent domain. The paper comes with a strong theoretical flavour and the proposed approach is well grounded.\nApplications in classic image generation tasks are provided.", "summary_of_the_review": "very interesting paper but with a very questionable english usage", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a simple approach called PDM for composing non-linear and complex normalizing flows with score-based generative models. Since score-based models can be considered as a special form of continuous-time normalizing flows, PDM corresponds to a composition of different classes of normalizing flows. 

Pros: 
* Combining generic normalizing flows with score-based models is an interesting direction as they have different characteristics and can be complementary to each other.
* Using Ito's lemma to show that the model learns a non-linear SDE in data space is valuable. 
* The authors show that the variational gap can be reduced using normalizing flows.

Cons: 
* The proposed method does not exhibit a clear advantage compared to the diffusion baseline without the normalizing flow component. On the CIFAR10 dataset, the best NLL and FID results are obtained by the diffusion baseline.

* Theorem 2 makes a very unrealistic assumption that a flow network is flexible enough to transform $p_r$ to any arbitrary distribution. If this holds, we wouldn't need the score-based generation model anymore. We could simply train the normalizing flow to map the input data distribution to a Normal distribution.

* This submission chooses to discuss differences with the recent LSGM framework. However, in doing so, several inaccurate claims are made. The lack of inference data diffusion in LSGM is mentioned as one of its drawbacks. However, it is not clear what is the value of having such a mechanism and what implications it may have on the expressivity of the model. Note that mapping from data space to latent space in VAEs can be considered as a stochastic inversion rather than an exact inversion. Ito's lemma does not require invertibility and it can be easily applied to the forward and generative diffusion in LSGM. The authors argue that applying it to the forward diffusion in LSGM will result in $\hat{p_{r}}\ne p_{r}$. But, $\hat{p_{r}}$ would be only considered for visualization of the forward diffusion and it is not used for training or any other purposes. LSGM, the proposed PDM, and score-based models are all trained with a reweighting of ELBO (see [here](https://arxiv.org/abs/2106.02808)). It is not clear if the drawback mentioned above has an impact on the training or expressivity of the model.

* The presentation in the paper requires improvement. The motivation on why invertibility plays a key role is not clear beyond generating the visualization in Figure 2. 

In summary, the paper proposes an interesting idea and explores directions very relevant to the current focus in generative learning. However, given the concerns above, we don't believe that the paper in its current form is ready for presentation at ICLR.
Decision: Reject