Diffusion Normalizing Flow
Authors: Qinsheng Zhang
Yongxin Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: normalizing flow
diffusion probabilistic models
density estimation
generative models

Abstract: We present a novel generative modeling method called diffusion normalizing flow based on stochastic differential equations (SDEs). The algorithm consists of two neural SDEs: a forward SDE that gradually adds noise to the data to transform the data into Gaussian random noise, and a backward SDE that gradually removes the noise to sample from the data distribution. By jointly training the two neural SDEs to minimize a common cost function that quantifies the difference between the two, the backward SDE converges to a diffusion process the starts with a Gaussian distribution and ends with the desired data distribution. Our method is closely related to normalizing flow and diffusion probabilistic models, and can be viewed as a combination of the two. Compared with normalizing flow, diffusion normalizing flow is able to learn distributions with sharp boundaries. Compared with diffusion probabilistic models, diffusion normalizing flow requires fewer discretization steps and thus has better sampling efficiency. Our algorithm demonstrates competitive performance in both high-dimension data density estimation and image generation tasks.

Diffusion Normalizing Flow
Qinsheng Zhang Georgia Institute of Technology gzhang4198gatech.. edu
Yongxin Chen Georgia Institute of Technology yongchen@gatech. edu
Abstract
We present novel generative modeling method called diffusion normalizing flow based on stochastic differential equations (SDEs). The algorithm consists of two neural SDEs: forward SDE that gradually adds noise to the data to transform the data into Gaussian random noise, and backward SDE that gradually removes the noise sample from the data distribution. By jointly training the neural SDEs to minimize a common cost function that quantifies the difference between the two, the backward SDE converges distribution and ends with the desired data distribution. Our method is closely related normalizing flow and diffusion probabilistic models and can be viewed a combination of the two. Compared with normalizing flow, diffusion normalizing flow is able to learn distributions with sharp boundaries. Compared with diffusion probabilistic models, diffusion normalizing flow requires fewer discretization steps and thus has better sampling efficiency. Our algorithm demonstrates competitive performance in both high-dimension data density estimation and image generation tasks.
diffusion process the starts with Gaussian
1 Introduction
Generative model is class of machine learning models used to estimate data distributions and sometimes generate new samples from the distributions [8, 35, 16, 37, 7]. Many generative models learn the data distributions by transforming latent variable with tractable prior distribution to the data space [8, 35, 32). To generate new samples, one can sample from the latent space and then follow the transformation to the data space. There exist large class of generative models where the latent space and the data space are of the same dimension. The latent variable and the data are coupled through trajectories in the same space. These trajectories serve two purposes: in the forward direction x + 2, the trajectories infer the posterior distribution in the latent space associated with a given data sample x, and in the backward direction the trajectories starting from the latent space. This type of generative model can be roughly divided into categories, depending on whether these trajectories connecting the latent space and the data space are deterministic or stochastic.
generates new samples by simulating
When deterministic trajectories are used, these generative models are known as flow-based models. The latent space and the data space are connected through an invertible map, which could either be realized by the composition of multiple invertible maps [35, 8, 20] or a differential equation [4, 14]. In these models, the probability density at each data point can be evaluated explicitly using the change of variable theorem, and thus the training can be carried out by minimizing the negative -ig-liillhood (NLL) directly. One limitation of the flow-base based model is that the invertible map parameterized by neural networks used in it imposes topological constraints on the transformation from z to x. Such limitation affects the performance significantly when the prior distribution on is simple unimodal distribution such as Gaussian while the target data distribution is well-separated multi-moda distribution, i.c., its support has multiple isolated components. In [6], is shown that there some fundamental issues of using well-conditioned invertible functions to approximate such complicated mulli-modal data distributions.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

When stochastic trajectories are used, the generative models are often known as the diffusion model [38]. In a diffusion model, a prespecified stochastic forward process gradually adds noise into the data to transform the data samples into simple random variables. A separate backward process is trained to revert this process to gradually remove the noise from the data to recover the original data distributions. When the forward process is modeled by stochastic differential equation (SDE), the optimal backward SDE [1] can retrieved by learning the score function [39, 40, 17, 2]. When the noise is added to the data sufficiently slow in the forward process, the backward diffusion can often revert the forward one reasonably well and is able to generate high fidelity samples. However, this also means that the trajectories have to be sufficiently long with large number of dineeiiccetizzttion steps, which leads to slow training and sampling. In addition, since the forward process fixed, the way noise is added is independent of the data distribution. As a consequence, the learned model may miss some complex but important details in the data distribution, as we will explain later.
In this work, we present new generative modeling algorithm that resembles both the flow-based models and the diffusion models. It extends the normalizing flow method by gradually adding noise to the sampling trajectories to make them stochastic. It extends the diffusion model by making the forward process from x to z trainable. Our algorithm is thus termed Diffusion Normalizing Flow (Diffllow). The comparisons and relations among Doffflow, normalizing flow, and diffusion models are shown in Figure I. When the noise in Diffflow shrinks to zero, DiffFlow reduces to a standard normalizing flow. When the forward process is fixed to some specific type of diffusion, DiffFlow reduces to a diffusion model.
X1-1
Xi-a
Backward Backward/Sampling
Forward Diiffusin
Network
Noise
XL
Xie
X,e1
Normslizing Flows
Diffusion Models
lofff
The schematic diagram for normalizing flows, diffusion models, and DiffFllw. In normal- izing flow, both the forward and the backward processes are deterministic. They are the inverse of each other and thus collapse into single process. The diffusion model has fixed forward process and trainable backward process, both are stochastic. In DiffFlow, both the forward and the backward processes trainable and stochastic.
Figure 1:
In DifiFlow, the forward and backward diffusion processes are trained simultaneously by minimizing the distance between the forward and the backward process in terms of the Kullback-Leibhe (KL) divergence of the induced probability measures [42]. This cost turns out to be equivalent to (see Section 3 for a derivation) the (amortized) negative evidence lower bound (ELBO) widely used in variational inference [21]. One advantage to use the KL divergence directly is that it be estimated with no bias using sampled trajectories of the diffusion processes. The KL divergence in the trajectory space also bounds the KI. divergence of the marginals, providing alternative method to bound the likelihood (see Section for details). To summarize, we have made the following contributions.
We propose novel density estimation model termed diffusion normalizing flow (DiftFlow) that extends both the flow-based models and the diffusion models. The added stochasticity in DiffFlow boosts the expressive power of the normalizing flow and results in better performance in terms of sampling quality and likelihood. Compared with diffusion models, DiffFlow is able to learn a forward diffusion process to add noise to the data adaptively and more efficiently. This avoids adding noise to regions where noise is not so desirable. The learnable forward process also shortens the trajectory length, making the sampling much faster than standard diffusion models (We observe 20 times speedup over diffusion models without decreasing sampling quality much).
We develop stochastic adjoint algorithm to train the DiffFlow model. This algorithm evaluates the objective function and its gradient sequentially along the trajectory. It avoids storing all the

---

intermediate values in the computational graph, making it possible to train DiffFlow for high- dimensional problems.
3. We apply the DiffFlow model to several generative modeling tasks with both synthetic and real datasets, and verify the performance of DiffFlow and its advantages over other methods.
Background
Below we provide brief introduction to normalizing flows and diffusion models. In both of these models, we use r = {x(t), < t S } to denote trajectories from the data space to the latent space in the continuous-time setting, and (xo. X ... xN
in the discrete-time setting.
Normalizing Flows: The trajectory in normalizing flows is modeled by a differential equation
X f(x, t, 0),
(1)
parameterized by 0. This differential equation starts from random x(0) = x and ends at x(T) = z. Denote by p(x(t)) the probability distribution of x(t), then under mild assumptions, it evolves following [4]
i Olog (xx)((x()) Öt -r( Of
(2)
0x
Using this relation (1) (2) one can compute the likelihood of the model at any data point x.
In the discrete-time setting, the map from to in normalizing flows is composition of collection
of bijective functions as = FN FN-1 ... F2 o Fi. The trajectory T = {xo, X1. ,XN) satisfies
.NN)
x; F. -1 0): = (x,,0)
(3)
for all i. Similar to Equation (2). based on the rule for change of variable, the log- likeliiod of any data samples X can be evaluated as
N
log ddet( OF. (xx) )-
p(xo) = logp(xx)
Σ
(4)
Öx,
Since the exact likelihood is accessible in normalizing flows, these models can be trained by minimiz- ing the negative log-likelihood directly.
Diffusion Models: The trajectories in diffusion models are modeled by stochastic differential equations. More explicitly, the forward process is of the form
dx f(x, t)dt + g())dw
(5)
where the drift term f Rdd x R - Rdd is a vector-valued function, and the diffusion coefficient
R + R is scalar function (in fact, g is often chosen to be a constant). Here w denotes the standard Brownian motion. The forward process is normally a simple linear diffusion process [38, 16). The forward trajectory can be sampled using (5) initialized with the data distribution. Denote by pF the resulting probability distribution over the trajectories. With a slight abuse of notation, we also pp to denote the marginal distribution of the forward process.
The backward diffusion from z x(T" x x00 is of the form
(6) is well-known that when s coincides with the score function V Vlggrr: and x(T) in the forward and backward processes share the same distribution, the distribution pg induced by the backward process (6) is equal to PF [1],[29, Chapter 13]. To train the score network s(x, t, 9), one can use the KL divergence between pr and pn an objective function reduce the difference between pr and pn- When the difference is sufficiently small, pr and should have similar distribution over x(0). and one then use the backward diffusion (6) sample from the data distribution.
dx = [ t) - (t)s(xx 1, 0))dt g(t)dw.
In the discrete setting, the trajectory distributions can be more explicitly written as
N
N
pr(T) = px Pr((xxx--), Pb(T) = Ps (x())) II [Px(xxIlx))
The KL divergence between pF and pn can be decomposed according to this expression (7). Most diffusion models use this decomposition, and meanwhile take advantage of the simple structure of the forward process (5), to evaluate the objective function in training [39, 40, 41].
3

---

3
Diffusion normalizing flow
We next present our diffusion normalizing flow models. Similar to diffusion models, the DiffFlow models also has forward process
dx= f(x,1, O)dt + g(t)dw,
(8)
and backward process
dx f(x,t, g² (t) t, 8)]dt g(t)dw.
(9)
The major difference is that, instead of being fixed linear function as in most diffusion models, the drift term f is also learnable DiffFlow. The forward process is initialized with the data samples at = 0 and the backward process is initialized with a given noise distribution at t = '. Our goal is to ensure the distribution of the backward process at time = 0 is close to the real data distribution. That is, we would like the difference between pn (x()) and pr(((0)) to be small.
To this end, we use the KL divergence between (T) and pF(T) over the trajectory space as the training objective function. Since
(10) for any 0 S T by data processing inequality, small difference between (T) and (r(r) implies small difference between pa (x())) and pp (x()) in terms of KL divergence (more details are included Appendix B).
LL(pp(x(t))lpe)
- KL(pp(r)
3.1 Implementation
In real implementation, we discretize the forward process (8) and the backward process (9) as
X,+1 x; + (x + of Vat,
(11)
X;
X
[fe+ (x++1)
Si+ (xx + 0 V A..
(12)
where 06 (), unit Gaussian noise, [t, the discretization time points, and At, = t++1 - t, is the step size at the i-th step. Here we have dropped the dependence on the parameter 0 to simplify the notation. With this discretization, the KL divergence between trajectory distributions becomes
N
(x (xxxxx;11
KL(pF(r) Eropp [log (ogpp(Xx)) Eyppr log (X
E Erope [log pB (x; pB(x; - (2.)
Lg
LN
1,
(13)
The term Lo in (13) is constant determined by entropy of the dataset as
E
(r = Xa<pF
[r (Xo)] =: (x))))(00))
Tapp
The term LN is easy to calculate since pD( is simple distribution, typically standard Gaussian distribution.
To evaluate L1:N-1, we estimate it over sampled trajectory from the forward process pp. For a given trajectory 7 sampled from (T). we need to calculate Pz(T) along the same trajectory. To this end, specific group of { is chosen such that the same trajectory can be reconstructed from the backward process. Thus, 5/ satisfies
(T)
Xi+1 + [ 1 Si+ (xi+ [
(14)
Ji+1VAL
Since 8. is Gaussian noise, the negative log-likelihood term pn (x: Xi+1) is equal (T))
after dropping constants (see more details in Appendix B). In view of the fact that the expectation of Σ. (6 (7))² remains constant, minimizing Equation (13) is equivalent to minimizing the following loss (see Appendix C for the full derivation):
= Ex-epl logpe(xN) Σ (T)
logpB(XN)
- (T))
(15)
where the last equality based on a reparameterization trick [21]. We can minimize the loss in Equation (15) with Monto Carlo gradient estimation as in Algorithm 1.

---

Algorithm Training
repeat
Xy Real data distribution
01.N ~ ((0,
Discrete timestamps: t
Sample: (x, hi=o based on of
Gradient descent step Vol- logPB(xN) + E 26, (T))
until converged
Algorithm Stochastic Adjoint Algorithm for Diftflow
1: Input: Forward trajectory {x:)/20
2:
al
((6) polxk]
dxN
2
UNN
3: ØL
20
4: for
= N. N 1, do
5:
8L
8L
+ 2
(T))
8x,
(71)
0x;
ix;
Ox:
8x;
2 8x;
6:
OL
+
(T))
+
0(6 (T))²
Forward pass Backpropagation
20
00
2
0x:
7: end for
Figure 2: Gradient Flowchart
3.2 Stochastic Adjoint method
One challenge in training DiffFlow is memory consumption. When naive backpropagation strategy used, the memory consumption explodes quickly. Indeed, differentiating through the operations of the forward pass requires unrolling networks N times and caching all network intermediate values for every step, which prevents this naive implementation of DiffFlow from being applied in high dimensional applications. Inspired by the adjoint method in Neural ODE [4], propose the adjoint dx; and stochastic adjoint algorithm that allows training the DiftFlow model with reduced
variable OL memory consumption. In this adjoint method, we cache intermediate states and, based on these intermediate states, reproduce the whole process, including of' 8 as well as fú, 8; exactly.
We note another similar approach [26] training SDEs caches random noise dw and further takes advantage of the pseudo-random generator to save memory for the intermediate noises, resulting in constant memory consumption However, the approach can not reproduce exact trajectories due to time discretization error and requires extra computation to recover dw from the pseudo-random generator. We found that our image experiments in Section 4, the cached (x,) consumes only about 2% memory out of all the memory being used during training. Due to the accuracy and acceptable memory overhead, the introduced stochastic adjoint approach a better choice for DiffFlow. We summarize the method in Algorithm 2 and Figure 2. We also include the Py Porch [34] implementation the supplemental material.
3.3 Time discretization and progressive training
Figure At, of Lg and L,
We propose two time discretization schemes for training DiffFlow: fixed timestamps La and flexible timestamps f.g. For fixed times- tamps, the time interval [0. 7] is discretized with fixed schedule t; = ( With such fixed time discretization over batches, we denote the loss function as Lg. Empirically, we found B = 0.9 works well. This choice of increases stepsize At, when the forward pro- cess approaches = and provides higher resolution when the backward process is close to Xg. We found such discretization gen- erates samples with good quality and high fidelity details. The choice of polynomial function is arbitrary; other functions of similar sharps may work as well.
10
20
30
50
5

---

In the flexiable timestamps scheme, we train different batches with different time discretization points. Specifically, & is sampled uniformly from the interval I[ {<=)) ) (x-1) "T] for each batch. We denote the training objective function with such random time discretization Lg. We empirically found such implementation results in lower loss and better stability when we conduct progressive training where we increase N gradually training progresses. In progressive training, we refine the forward and backward processes as N increase. This training scheme can significantly save training time compared with the other method that uses a fixed large N during the whole process. Empirically, we found that progressive training can speed up the training up to 16 times.
To understand why such random time discretization scheme is more stable, we hypothesis that this choice encourages smoother f, since it seeks functions f, s to reduce objective loss under different sets of {4i} instead of a specific (4:) We illustrate fixed timestamps in L,g and a realization of random discretization in Lg in Figure 3 with 3 = 0.9.
3.4 Learnable forward process
0:337
t: 0.867
0.333
0.667
T
DitFloww
DDPM
FFFORD
Figure Illustration of forwarding trajectories of DiffFlow, DDPM, and FFJORD. Each row shows two trajectories of transforming data distributions, four rings, and Olympics rings, to a base distribution. Different modes of densities are in different colors. Though FFJORD adjusts forward process based on data, its bijective property prevents the approach from expanding density support to the whole space. DDPM can transform data distributions into Gaussian distribution but a data- invariant way of adding noise can corrupt the details of densities, e.g., the densities at the intersections of the rings. DiffFlow not only transforms data into base distribution but also keeps the topological information of the original datasets. Points from the same ring are transformed into continental plates instead of being distributed randomly.
The forward process not only is responsible for driving data into latent space, but also provides enough supervised information to learning backward process. Thanks to bijective property, NFs can reconstruct data exactly but there is no guarantee that it can reach the standard Gaussian. At the other extreme, Denoising diffusion probabilistic models (DDPM) [17] adopt a data-invariant forward diffusing schema, ensuring that xx is Gaussian. DDPM can even reach Gaussian in one step with N = 1, which output noise disregarding data samples. However, backward process will be difficult to Iearn if data is destroyed in one step. Therefore, DDPM adds noise slowly and often needs more than one thousand steps for diffusion.
The forward module of DiffFlow is a combination of normalizing flow and diffusion models. We show the comparision in fitting toy 2D datasets in Figure 4. We are especially interested in data with well-separated modes and sharp density boundaries. Those properties are believed to appear in various datasets. As stated by manifold hypothesis [36], real-world data lie on low-dimensional manifold [28] embedded in a siggh-imensional space. To construct the distributions in Figure 4, we rotate the 1-d Gaussian distribution (11,, 0.0012) around the center to form ring and copy the rings to different locations.
As a bijective model, FFJORD [14] struggles to diffuse the concentrated density mass into a Gaussian distribution. DiffFllw overcomes expressivity limitations of the bijective constraint by adding noise. As added noise shrinks to zero, the DiffFllow has no stochasticity and degrades to a flow-based model. Based on this fact, we present the following theorem with proof in Appendix A.

---

Theorem 1.
As diffiusion coefficients - O, DiffFllw reduces to Normalizing Flow. Moreover, minimizing the objective function in Equation (13) is equivalent minimizing the negative log- likelihood as in Equation (4).
DDPM [17] uses fixed noising transformation. Due the data-invariant approach p(xr|xx) p(XT), points are diffused in the same way even though they may appear in different modes or different datasets. We observe that sharp details are destroyed quickly in DDPM diffusion, such as the intersection regions between rings. However, with the help of learnable transformation, DiffFlow diffuses in much efficient way. The data-dependent approach shows different diffusion strategies on different modes and different datasets. Mcanwhile, similar to NFs, it keeps some topological information for learning backward processes. We include more details about the toy sample in Section 4.
4 Experiments
We evaluate the performance of DiffFlow in sample quality and likelihood on test data. To evaluate the likelihood, we adopt the marginals distribution equivalent SDEs
1 + X²
dx = [( t,
2 g²(t)s(x,t, t, f)]tt + Ag(t)dw,
(16)
with A 2 0 (Proof see Appendix H). When À = O, it reduces to probability ODE [41]. The ODE provides efficient way evaluate the density and negative logelikeli.ood. For any 0 S À S the above SDE can be used for sampling. Empirically, we found À = 1 has the best performance.
4.1 Synthetic 2D examples
We compare the performance of DiffFlow and existing diffusion models and NFs on estimating the density of 2-dimensional data. We compare the forward trajectories of DiffFlow, DDPM [17] and FFJORD [14]' in Figure 4 and its sampling performance in Figure 5. To make a fair comparison, we build models with comparable network sizes, around 90k learnable parameters. We include more training and model details in Appendix E.
All three algorithms have good performance on datasets whose underlying distribution has smooth density, such as 2 spirals. However, when we shrink the support of samples or add complex patterns, performance varies significantly. We observe that FFJORD leaks many samples out of the main modes and datasets with complex details and sharp density exacerbates the disadvantage.
DDPM has higher sample quality but blurs density details, such as intersections between rings, areas around leaves of the Fractal tree, and boxes in Sierpiński Carpet. The performance is within expectation given that details are easy to be destroyed and ignored with the data-invariant noising schema. On the less sharp dataset, such as 2 Spirals and Checkerboard, its samples align with data almost perfectly.
DiffFlow produces the best samples (according to a human observer). We owe the performance the flexible noising forward process. As illustrated in Figure 4, DiffFlow provides more clues and retains detailed patterns longer for learning its reverse process. We also report a comprehensive comparison of the negative likelihood and more analysis in Appendix E. DiffFllw has much lower negative likelihood, especially on sharp datasets.
4.2 Density estimation on real data
We perform density estimation experiments on five tabular datasets [33]. We employ the probability flow to evaluate the negative log-likelihood. We find that our algorithm has better performance in most datasets than most approaches trained by directly minimizing negative log- oog-liillhood, including NFs and autoregressive models. DiffFlow outperforms FFJORD by wide margin on all datasets except HEPMASS. Compared with autoregressive models, it excels NAF [18] on all but GAS. Those models require (dl) computations to sample from. Meanwhile, Diffflow is quite effective in achieving such performance with MLPs that have less than 5 layers. We include more details in Appendix F
4.3 Image generation
In this section, we report the quantitative comparison and qualitative performance of our method and existing methods on common image datasets, MNIST [24] and CIFAR-10 10 [23]. We use the same
Implementation of FFJORD and DDPM are based on official codebases.
7

---

Slarr Olympiic
Olyusic
Chuckehund
Prarial Tow
Suepidaki Carpet
Spirala
Duio
DIPA
FFIORD
Figure 5: Samples from DifiFlow, DDPM and FFJORD on 2-D datasets. All three models have reasonable performance datasets that have smooth underlying distributions. But only DiffFlow is capable to capture complex patterns and provides sharp samples when dealing with more challenging datasets.
Dataset
POWER
GAS
HEPMASS
MINIBOONE
BSDS300
RealNVP [8]
-0.17
-8.33
18.71
13.55
-153.28
FFJORD [14]
-0.46
-8.59
14.92
10.43
-157.40
DiffFlow (ODE)
-1.04
-10.45
15.04
8.06
-157.80
MADE [11]
3.08
-3.56
20.98
15.59
-148.85
MAF [33]
-0.24
-10.08
17.70
11.75
-155.69
TAN [31]
-0.48
-11.19
15.12
11.01
-157.03
NAF [18]
-0.62
-11.96
15.09
8.86
-157.73
Table I: Average negative log-likelihood (in nats) on tabular datasets [33] for density estimation (lower better).
unconstrained U-net style model used successfully Ho et al. [17] for drift and score network.
We reduce the network size to half of the original DDPM network so that the total number of trainable parameters of DiffFllw and DDPM are comparable. We small N 10 the beginning of training and slowly increase to large N as training proceeds. The schedule N reduces the training time greatly compared with using large N all the time. We use constants 9i and 0.05 for MNIST and CIFARIO, and N = 30 for sampling MNIST data and N = 100 for sampling CIFARIO. As it is reported by Colicoeur-Martinent
al. [19], adding noise the last step will significantly lower sampling quality, we use one single denoising step at the end of sampling with Tweedie's 's formula [10].
We report negative log- likelihooh (NLL) in bits per dimension or negative ELBO if NfL is unavailable. On MNIST, we achieve competitive performance on NLL as in Table 2. We show the uncurated samples from DiffFlow in Figure and Figure On CIFAR-IO. DiffFlow also achieves competitive NLL performacce shown in Table 3. DiffFlow performs better than normalizing flows and DDPM models, but is slightly worse than DDPM++(sub, deep, sub-vp) and Improved DDPM. However, these approaches conduct multiple architectural improvements and use much deeper and wider networks. also report the popular sample metric, Fenchel Inception Distance (FID) [15]. DiffFLow has lower FID score than normalizing flows and has competitive performance compared with DDPM trained with unweighted variational bounds, DDPM and Improved DDPM. is worse than DDPM trained with reweighted loss, DDPM (L.), DDPM cont, and DDPM++ [17, 41, 30). Besides, sampling

---

Figure 6: MNIST Samples
Figure 7: CIFAR10 Samples
Table 2: NLL MNIST
0
3 1
2
2
7
/
4
6
4
9
Model
NLL (L)
2
7
3
2
3
4
4
8
RealNVP [8]
1.06
0
1
0
Glow [20]
1.05
4
2
7
FFJORD [14]
0,99
2
9
2
2
0
8 5
ResFlow [5]
0.97
2
4
5
2
DiffFlow 0.93
7
1
7 P
3
1
6
3
0
2
4
0
8
Q
quality with different sampling steps N are compared Table 4 The advantage of DiffFlow is clear when we compare relative FIDs degeneracy ratio with N = 100 respectively. DiffFlow is able retain better sampling quality when decreasing N. Full details on architectures used, training setup details, and more samples can be found in Appendix G.
Table 3: NLLs and FIDs on CIFAR-10.
Table 4: FIDs with various N
Model
NLL())
FID (L)
N DiffFloWDDPM (Z) DDPM (I,) DDIM
ReaINVP [8]
3.49
28.31
373.51
370.23
44.69
Glow [20]
3.35
46.90
10
22.56
364.64
365.12
18.62
Flow++ [16]
3.29
20
17.98
138.84
135.44
10.89
FFJORD [14]
3,40
50
14.72
47.12
34.56
7.01
ResFlow [5]
3.28
46.37
100
13.43
22.23
10.04
5.63
DDPM (L) [17]
< 3.70
13.51
DDPM (L.) [17]
< 3.75
3.17
Table 5: Relative FIDs degeneracy ratio
DDPM (L.)ODE) [41]
3.28
3.37
DDPM cont. (sub-V VBb- [41]
3.05
3.56
N
DiffFlowDDPM (L) DDPM (L,) DDMM
DDPM++ (sub-VP) [41]
3.02
3.16
DDPM++ (deep, sub-VP VP) [41]
2.99
2.94
5
2.12
16.80
37.02
7.94
Improved DDPM [30]
<2.94
11.47
10
1.68
16.40
36.12
3.31
DiffFlow (Lg)
3.71
13.87
20
1.34
6.24
13.54
1.93
DiffFlow (L.)
3.67
13.43
50
1.10
2.12
3.45
1.24
1.0
1.0
1.0
1.0
DiftFlow (Lg. ODE)
3.04
14.14
5 Related work
Normalizing flows [8, 35] have recently received lots of attention due to its exact density evaluation and ability to model high dimensional data [20, 9]. However, the bijective requirement poses limitations modeling complex data, both empirically and theoretically [42, 6]. Some works attempt to relax the bijective requirement; discretely index flows [9] use domain partitioning with only locally invertible functions. Continuously indexed flows [6] extend discretely indexing to a continuously indexing approach. As pointed out in Stochastic Normalizing Flows (SNF) [42], stochasticity effectively improve the expressive power of the flow-based model in low dimension applications. The architecture used in SNF, which requires known underlying energy models, presents challenges for density learning tasks; SNF is designed for sampling from unnormalized probability distribution instead density estimation. Besides, even with ideal networks and infinite amount of data, due to the predefined stochstic block being used, SNF cannot find models with aligned forward and backward distribution as DifiFllow
When it comes stochastic trajectories, minimizing the distance between trajectory distributions has been explored existing works. Denoising diffusion model [38] uses fixed linear forward diffusion schema and reparameterizes the KI. divergence such that minimizing loss is possible without computing whole trajectories. Diffusion models essentially corrupt real data iteratively and learn to remove the noise when sampling. Recently, Diffusion models have shown the capability to model
"The performance of DDPM is evaluated based on the officially released checkpoint with L. denotes for Tesimple in the original paper.

---

high-dimensional data distribution, such as images [17, 40), shapes [3], text-to-speede [22]. Lately, the Score- model [41] provides a unified framework for score-matching methods and diffusion models based on stochastic calculus. The diffusion processes and sampling processes can be viewed as forwarding SDE and reverse-time SDE. Thanks to the linear forward SDE being used in DDPM, the forward marginal distributions have closed-form and are suitable for training score functions on large-scale datasets. Also, due to the reliance on fixed linear forward process, it takes thousands of steps to diffuse data and generate samples. DiffFllw considers general SDEs and nosing and sampling are more efficient.
Existing Neural SDE approaches suffer from poor scaling properties. Backpropagating through solver [12] has a linear memory complexity with the number of steps. The pathwise approach [13] scales poorly in computation complexity. Our stochastic adjoint approach shares a similar spirit with SDE adjoint sensitivity [25]. The choice of caching noise requires high resolution of time discretization and prevents the approach from scaling to high dimension applications. By caching the trajectory states, DiffFlow can use a coarser discretization and deploy on larger dimension problems and problems with more challenging densities. The additional memory footprint is negligible compared with the other network memory consumption in DiffFlow.
6 Limitations
While DiffFlow gains more flexibility due to the introduction of learnable forward process, it loses the analytical form for (x( and thus the training less efficient compared with acore-based loss [41]. Training DiffFlow relies on backpropagation through trajectories and is thus significantly slower than diffusion models with affine drift. Empirically,
found DiffFlow is about 6 times slower than DDPM in 2d toy examples, 55 times in MNIST, and 160 times in CIFARIO without progressive training in Section 3.3. Though the stochastic adjoint method and progressive training help save memory footprint and reduce training time, the training of DiffFlow is still more expensive than DDPM and its variants. On the other hand, compared with normalizing flows, the extra noise in DiftFlow boosts the expressive power of the model with little extra cost. Though DiffFlow trained based on SDE, its marginal distribution equivalent ODE 2 shows much better performance than its counterpart trained with ODE [14]. It is interesting to investigate, both empirically and theoretically. the benefit in terms of expressiveness improvement caused by stochastic noise for training normalizing flows.
7 Conclusions
We proposed novel algorithm, the diffusion normalizing flow (DiffFlow), for generative modeling and density estimation. The proposed method extends both the normalizing flow models and the diffusion models. Our DiffFlow algorithm has two trainable diffusion processes modeled by neural SDEs, one forward and one backward. These two SDEs are trained jointly by minimizing the KL divergence between them. Compared with most normalizing flow models, the added noise in DiffFlow relaxes the bijectivity condition in deterministic flow-based models and improves their expressive power. Compared with diffusion models, DiffFlow learns more flexible forward diffusion that is able to transform data into noise more effectively and adaptively. In our experiments, we observed that DiffFlow is able model distributions with complex details that are not captured by representative normalizing flow models and diffusion models, including FFJORD, DDPM. For CIFAR 10 dataset, our DiffFlow method has worse performance than DDPM in terms of FID score. We believe our Doffflow algorithm can be improved further by using different neural network architectures, different time discretizing method and different choices of time interval. We plan explore these options in the near future.
Our algorithm is able to learn the distribution of high -ich-innnnionaal data and then generate new samples from it. Like many other generative modeling algorithms, it may be potentially used to generate misleading data such fake images videos.
Acknowledgements
The authors would like to thank the anonymous reviewers for useful comments. This work is supported in part by grants NSF CAREER ECCS-1942522 and NSF CCF-2008513,
10

---

References
[1] Anderson, B. D. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12 (3):313-326, 1982.
[2] Bordes, F., Honari, S., and Vincent, P. Learning to generate samples from noise through infusion training. arXiv preprint arXiv: 06975, 2017.
[3] Cai, R., Yang, G., Averbuch-Floor H., Hao, Z., Belongie, S., Snavely, N., and Hariharan, B. Learning gradient fields for shape generation. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[4] Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural ordinary differential equations. arXiv preprint arxiv: /800:000667666 2018.
[5] Chen, R. Q., Behrmann, J. Duvenaud, D., and Jacobsen, Residual flows for invertible generative modeling. In Advances in Neural Information Processing Systems, 2019.
[6] Cornish, R., Caterini, A.. Deligiannidis, G., and Doucet, A. Relaxing bijectivity constraints with contin- uously indexed normalising flows. In International Conference on Machine Learning, pp. 2133-2143. PMLR, 2020.
Deng. Y., Bakhtin, A., Ou, M., Szlam, A., and Ranzato, M. Residual energy-based models for text generation. arXiv preprint arXiv:2004.1177 11714, 2020.
[8] Dinh, L. Sohl-Dickstein, J., and Bengio, S. Density estimation using real nvp. arXiv preprint arXiv: 008803, 2016.
[9] Dinh, L., Sohk-DDckstein, J., Larochelle, H., and Pascanu, R. A rad approach to deep mixture models. arXiv preprint arXiv:1903.0m 1903,097137.
2019. 2019.
[10] Efron, B. Tweedie's formula selection bias. Journal of American Statistical Association, 106(496):
1602-1614, 2011.
[11] Germain, M., Gregor, K., Murray, I., and Larochelle, H. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pp. 881-889. PMLR, 2015.
[12] Giles, M. and Glasserman, P. Smoking adjoints: Fast monte carlo greeks. Risk, 19(1): 2006.
[13] Gobet, E. and Munos, R. Sensitivity analysis using itô-malliainn calculus and martingales. and application to stochastic optimal control. SIAM Journal on control and optimization, 43(5):1676-1 1713, 2005.
[14] Grathwohl, Chen, T., Bettencourt, Sutskever, I., and Duvenaud, Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint 1810.001 2018.
[15] Heusel, M., Ramsauer. H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained two time-scale update rule converge to local nash equilibrium. arXiv preprint S..iv:1706.0555550 2017.
[16] Ho, J., Chen, X., Srinivas, A., Duan, Y., and Abbeel, P. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference Machine Learning, pp. 2722-2730. PMLR, 2019.
[17] Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. arXiv preprint arXiv: 2020.
[18] Huang, C.-W., Krueger, D., Lacoste, A., and Courville, A. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2083 PMLR, 2018.
[19] Martinccu,,, A., Piché- Taillefer, R., Combes, R. T. d., and Mitliagkas, Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.05475, 2020.
[20] Kingma, D. and Dhariwal, Glow: Generative flow with invertible Ixl convolutions. arXiv preprint arXiv:1807, 1807.0009,, 2018.
[21] Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[22] Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009. 09761, 2020.
[23] Krizhevsky, A., Hinton, G., Learning multiple layers features from tiny images. 2009.
[24] LeCun, Y. The mnist database of handwritten digits. http://yann. lecun. com/exdh/mnisst, 1998.
[25] Li, X., Wong, T.-K. L., Chen, R. T.,, and Duvenaud, D. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, 3870-3882, PMLR, 2020.
[26] Li. X., Wong, T.-K. L., Chen, R. T., and Duvenaud, D. Scalable gradients for stochastic differential equations. In International Conference on Artificial Intelligence and Statistics, 3870-3882. PMLR, 2020.

---

[27] Meng. C., Song, J., Song, Y., Zhao, S., and Ermon, S. Improved autoregressive modeling with distribution smoothing. arXiv preprint arXiv:2103. 15089, 2021.
[28]
Narayanan, H. and Mitter, S. Sample complexity of testing the manifold hypothesis. In Proceedings of the 23rd International Conference on Neural Information Processing Systems- Volume 2. pp. 1786-1794 2010.
[29] Nelson, E. Dynamical theories of Brownian motion. Princeton university press. 2020.
[30] Nichol, A. and Dhariwal, P. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.099 09672, 2021.
[31] Oliva, J., Dubey, A., Zaheer, M., Poczos, B., Salakhutdinov, R., Xing. E., and Schneider, J. Transformation autoregressive networks. In International Conference on Machine Learning, 3898-3907. PMLR, 2018.
[32] Oord, v. d.. Dieleman. S., Zen, H., Simonyan. K.. Vinyals, O., Graves, A., Kalchbrenner, N.. Senior, A.. and Kavukeuogla, K. Wavenet: generative model for raw audio, arXiv preprint arXiv: 1609.03499, 2016.
[33] Papamakarios, G., Pavlakou, T., and Murray, I. Masked autoregressive flow for density estimation. arXiv preprint /rxiv:/75.00705 2017.
[34] Paszke, A., Gross, S., Massa, E,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N.,
Antiga, I.., al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: arXiv:/912.0f 19t2.0112.. 2019.
[35] Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In International Conference Machine Learning, pp. 1530-1538. PMLR, 2015.
[36] Roweis, S. and Saul, L. K. Nonlinear dimensionality reduction by locally linear embedding. science,
29005500):232332236, 2000.
[37] Salimans, T., Karpathy, A., Chen, X., and Kingma, D. P. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.0551/1 2017.
[38] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli. S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp. 2256 2265. PMLR, 2015.
[39] Song. Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895- 11907, 2019.
[40] Song. and Ermon, S. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006. 0901 2020.
[41] Song. Y., Sohl- Dickstein, Kingma, D., Kumar, A., Ermon, S., and Poole, B. Score-based generative mod- eling through stochastic differential equations. In International Conference on Learning Representations, http//ormm//PPPI111228BB
2021. URL /ttpeee//pperrevie
[42] Wu, H., Köhler, J., and Noe, Stochastic normalizing flows. In Larochelle, H., Ranzato, M., Hadsell, Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Prucessing Systems, volume 33, pp. 5933-5944. Curran Associates, Inc., 2020.
12


Reviews:
[{"review_id": "gwW4MrbPYcl", "reviewer": "Reviewer_H9qy", "summary": "This paper proposes to lean a more expressive forward process in a diffusion model by using a neural network drift function.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think it is a nice insight that diffusion models can be thought of as relaxed normalizing flows. The method to learn a more expressive forward process is new, although it is straightforward and does not really resolve key issues in learning more expressive forward processes. For example, it is well known that the affine forward processes allow for sampling of $q(x_t | x_0)$ for arbitrary $t$, allowing for fast training by picking random $t$; furthermore, variance of training and evaluation can be reduced by Rao-Blackwellizing the variational lower bound into $x_0$ MSE prediction losses. Both properties do not seem true with the presented method when f is a neural network, and the computational complexity tradeoffs should be studied more thoroughly (e.g. what is the quantitative benefit of slower training time vs better sample quality or log likelihood?)\n\nWhile it is interesting to learn more expressive forward processes, it is not clear why the method needs to be presented as a combination of diffusion and normalizing flows: it seems clear enough to present the method as learning a more expressive forward process for diffusion models. Also, I found it difficult to understand the description of time discretization in Section 3.3.\n\nIn Section 4.1, I am surprised that DDPM \"blurs density details.\" Shouldn't it be possible to resolve this by carefully tuning the noise schedule? What hyperparameters were used for this experiment?\n\nI find Table 4 slightly misleading. It is true that the FID scores are better than DDPM for small N, but for large N, the FID scores of DDPM are around 3, and it seems that the presented method does not approach that score.\n\nWhy does the table omit log likelihoods and FID scores from published methods that perform better, e.g. from Score-Based Generative Modeling through Stochastic Differential Equations by Song et al? The paper claims state-of-the-art NLL (line 246), but it is not.\n\nThe main limitation of the paper is that as it is presented, the results do not significantly improve upon the baseline enough to warrant the more complex training procedure and model construction, there is no adequate study of the training time (or FLOPS) vs NLL or sample quality, and in some cases, the results are presented in a slightly misleading way (e.g. Table 4).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations of the paper are not adequately addressed, see the main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "INQgZEElIDS", "reviewer": "Reviewer_i723", "summary": "This paper proposes \"Diffusion Normalizing Flow\", which generalizes current diffusion-based models to have a trainable inference process. This allows the model to transform structured data into an unstructured prior in a more efficient manner, thus resulting in faster sampling.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Significance and Limitation\n\nThe paper serves as a natural extension of [1] and [2], which have a fixed inference process that slowly transforms an arbitrary data distribution into an unstructured prior and learn to revert the dynamic. This paper proposes to parameterize the inference part so that the transformation of the data into an unstructured noise can be more efficient (as illustrated by Fig 4). \n\nTwo limitations of the proposed methods:\n\n1. Computational time: Despite the generality this new framework provides, I suspect training will take much more time since one would have to backprop through the entire trajectory. This is in contrast to the proposal of a recent concurrent work [3], which, due to the constraint on the form of the inference process, still allows exact sampling of the perturbed data at any time step. I see this as a limitation since having to backprop through the entire trajectory means it will make it harder to scale up to larger datasets. \n2. Memory: Backpropagation through the entire path also raises another problem — the increase in memory. This work proposes an \"adjoint\" method and claims to have constant memory consumption (in the number of integration steps). However, I found it to be not entirely true, as the algorithm requires caching all intermediate states $x_i$, which means it is still technically $O(N)$. A real constant-memory adjoint method would not require storing all the intermediate states, such as neural ODE, and another recent work [4], which suggests using the stochastic adjoint sensitivity method proposed by [5]. \n\n[1] Deep unsupervised learning using nonequilibrium thermodynamics\n\n[2] Denoising diffusion probabilistic models\n\n[3] Variational Diffusion Models\n\n[4] A Variational Perspective on Diffusion-Based Generative Models and Score Matching\n\n[5] Scalable gradients for stochastic differential equations\n\nPresentation and clarity also need some work; see the following points for more detail. \n\n### More technical questions and criticisms:\n\n1. I don't think we can simply drop all the constants as claimed in L142. For example, the normalizing constants of the Gaussians will depend on the variance, which is the diffusion coefficients of the SDEs, unless these coefficients are either constant or not trained. Overall I find the derivation in Section 3.1 very confusing; perhaps having a detailed derivation in the appendix will help.  \n2. Is there a runtime comparison between DDPM and the proposed method in training? e.g. how much time does it take to make one parameter update, fixing the model size? \n3. It is claimed that DiffFlow is a combination of flows and diffusion models (e.g. L179). It's not very clear to me how NF is integrated into this framework. Do you compose flows with diffusion models? Or it's just a matter of interpretation?\n4. Aside from the use of the equivalent ODE for density evaluation (which is originally proposed by [6], is there other reasons to introduce the more general marginal-equivalent SDEs, i.e. eq (17)? Also, why not report the ELBO of the SDE?\n5. What is the concept of \"layer\" within this paper? e.g. in L231, it says DiffFlow uses no more than 5 layers. Is it the size of the network that parameterizes f, g, and s? \n6. The statement of Theorem 1 is very handwavy, and so is the proof. \"is equivalent to\", under what condition? In the proof it seems it requires fixing $p_B(x_i|x_{i-1})=p_F(x_i|x_{i-1})$, what does it mean by it's possible when g is small, e.g. a Gaussian approximation will suffice? It doesn't feel like a rigorous proof. \n\n[6] Score-based generative modeling through stochastic differential equations\n\n### Typos, and other suggestions:\n\n- L36: I'd put \"continuous-time\" normalizing flows, instead of just normalizing flows\n- L100: $f:\\mathbf{R}^d\\times \\mathbf{R}\\rightarrow \\mathbf{R}^d$\n- L103 $p_F$ is defined to be the distribution over trajectories, but then in L106 it's used as the score, which is the gradient of the marginal density. This is not accurate.\n- Eq (10): why does it hold? Is it the data processing inequality? Please clarify.\n- L192: noising transformation?\n- In Appendix A, [39] is cited, but is not displayed in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "HLKDCR7vX6M", "reviewer": "Reviewer_Sbrg", "summary": "The paper under review introduce the Diffusion Normalizing Flow, which combines the diffusion model method and the normalizing flow method in their algorithm. The idea is clearly presented and the main part of the paper is focused on the experiment implementation of the algorithm. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive.\n\n1. The idea seems to be new. Indeed, the proposed DiffFlow is one type of the Neural SDEs, where the volatility only depends on time variable. \n\n2. The idea and the comparison to diffusion model and normalizing flow is clearly presented in the figure and one theoretical result is provided to link the three different methods.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Negatives.\n\n1. The model itself seems to lack theoretical justification. To be precise, The proposed diffusion model with equation (5), (6) does not come naturally from the cited references [34,16]. It seems that the original form of the backward diffusion process is first introduced in [37]. This is not clearly stated, if the reviewer did not miss other key references. \n\n2. Furthermore, the score function $\\mathbf s(x,t,\\theta)$ appears in the backward process without derivation or justification. The one in [37] follows from the results in Anderson 1982.  The reviewer is curious about the original sources of equation (6), or can the author provides the derivation of this backward process.  The term \"stochastic adjoint\" is used in the paper, but the adjoint process is not well explained in the paper, together with its link to equation (6).\n\n3. Once equation (5),(6) are introduced, the author then claim to consider the DiffFlow by letting $f(t,x)$ to be $f(t,x,\\theta).$ Is there any reason to do so? Why the diffusion function $g(t)$ is fixed in this case. Does this change keep the property of the original backward process? From the reviewer's point of view, the algorithm is rather intuitive, is there any derivation of the backward or adjoint process here after $f(t,x,\\theta)$ being used.\n\n4. It seems that the algorithm is very similar to the recent paper [Efficient and Accurate Gradients for Neural SDEs]. Can the author do some comparison in this case? (e.g. memory, efficiency, etc..)\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "GAqdYsxLoXv", "reviewer": "Reviewer_b7rB", "summary": "The paper introduces a non-linear drift component to the SDE formulation of diffusion-based generative modelling. Earlier work (DDPM) used a fixed linear drift component. This paper uses a trainable neural network as the drift component, and another one to estimate the score. Both these neural networks are trained by minimizing the KL divergence between the forward and backward trajectories, while accounting for their SDE formulations. It is found that this trained network performs reasonably well compared to prior works, and achieves better sample quality and shorter trajectories compared to DDPM as well as Continuous Normalizing Flows (FFJORD).", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clear enough to understand, although there are some typos in a few places (mentioned below).\n\nThe \"novel\" contribution of making the drift component non-linear is relevant in the context of this line of research : discrete Markov chains (DDPM [1], etc.), to equivalent continuous formulations with the help of SDEs [2], to a variety of SDEs where the drift component is learnt as well instead of being prefixed.\n\nRelevant examples have been shown in 2D data as well as image sampling to highlight the advantages of this method compared to the previous version (DDPM). Although the continuous formulation (SDE) is mentioned, the paper converts it back to a discrete Markov chain and trains it accordingly.\n\nFigures 5, 8 and 9 illustrate the relevance of the current method compared to DDPM and FFJORD. They make a convincing argument for the use of a trainable forward process. Lines 182-184 make a convincing argument for why this particular type of 2D data was chosen. It is all the more encouraging that even though finer details are better, it is not at the expense of likelihood/NLL (in many cases).\n\nA major significance is that the number of sampling steps can be quite low compared to the original formulation (Table 4) and yet the quality of generated samples does not significantly deteriorate as much as the baseline (DDPM).\n\nTable 3 compares the NLL and FID of previous relevant methods for CIFAR10. DDIM [3] is missing from the list, as it is also a discrete formulation like DDPM but uses a different sampling scheme. It is a relevant comparison since DDIM also aims to reduce the number of training steps. Although, the method proposed is significantly different from the motivation of DDIM, in that DDIM preserves the DDPM formulation and only changes the sampling scheme, while the paper changes the forward process itself. \n\nThe appendix mentions the resources used to complete each experiment. It needs to be seen whether computational efficiency could be improved.\n\nCode has been provided to reproduce the experiments.\n\n[1] Denoising Diffusion Probabilistic Models, NeurIPS 2020\n[2] Score-based generative modeling through stochastic differential equations, ICLR 2021\n[3] Denoising Diffusion Implicit Models, ICLR 2021\n\nLine 87 says x(0) = z and x(T) = x, but lines 121-122 say x(T) = x and x(0) = z. I assume the latter, it is good if this is clarified.\n\nLine 55 : the forward process from x* to z* ?\n\nLine 166 : (to add) in Figure 3*\n\nLine 192 : noi*sing\n\nLine 198 : to*pological?\n\nFigure 5 : Frac*tal tree\n\nLine 239 : using*\n\n---\n\nUpdate : I stand by my original evaluation of accepting this paper:\n\nThe \"novel\" contribution of making the drift component non-linear is relevant in the context of this line of research\nI found their methodology to be well motivated\nThe pros and cons of each design step have been mentioned\nEnough experiments have been performed to explore the method, with the pros and cons\nCode has been provided for reproduceability\nComputational resources used for each experiment have been mentioned\nReviewers' comments have been responded to sufficiently and honestly\nIf the other reviewers feel there are still some unanswered questions, or comments that have not been responded to, please let us know. I think this paper deserves to be accepted.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The appendix describes the computational resources used in training the models. The figures and tables clearly illustrate the limitations of the method. It could be mentioned that lower sampling steps could alleviate computational burden significantly.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was significant discussion of this paper, with one reviewer increasing their score. The score range was unusually wide, even after discussion.

Reviewers felt that the idea presented was both novel and interesting. I agree with this assessment -- I think this paper draws a very nice and interesting link between normalizing flows and diffusion models. It's impressive to cast them into the same theoretical framework, and I think this will inspire followup work.
However, reviewers also had remaining concerns. Most seriously that:
- the computational tradeoffs during training that this approach required weren't clearly acknowledged or discussed. I believe this is a fair criticism. Please include discussion of this, and the training time tradeoffs you discussed in your rebuttal, in any camera ready.
- that the memory discussion around the adjoint method is misleading. I am unable to fully judge this concern. I do think something like logarithmic checkpointing should also be able to prevent memory overhead from being an issue though, so it's not clear to me that memory usage was ever a critical problem to start with.
- that the performance table left out a number of results with better FID and log likelihood scores than the presented method. This is definitely true, and the additional rows **must** be added for any camera ready. The paper only claimed "competitive performance", not SOTA, so I do not believe this is a fatal flaw.
- there were also some concerns about clarity, and I encourage the authors to address the specific aspects that reviewers found challenging.

Due to these legitimate concerns, I struggled with my recommendation for this paper, and read the paper myself to make a more informed decision. Because of the novelty of the approach, and because of my judgement of the relative importance of novelty to scientific progress, I am recommending acceptance.
Decision: Accept (Poster)