Diffusion Models with Deterministic Normalizing Flow Priors
Authors: Mohsen Zand
Ali Etemad
Michael Greenspan
Venue: Accepted by TMLR ()

Abstract: For faster sampling and higher sample quality, we propose DiNof ($\textbf{Di}$ffusion with $\textbf{No}$rmalizing $\textbf{f}$low priors), a technique that makes use of normalizing flows and diffusion models. We use normalizing flows to parameterize the noisy data at any arbitrary step of the diffusion process and utilize it as the prior in the reverse diffusion process. 
More specifically, the forward noising process turns a data distribution into partially noisy data, which are subsequently transformed into a Gaussian distribution by a nonlinear process. 
The backward denoising procedure begins with a prior created by sampling from the Gaussian distribution and applying the invertible normalizing flow transformations deterministically. To generate the data distribution, the prior then undergoes the remaining diffusion stochastic denoising procedure.
Through the reduction of the number of total diffusion steps, we are able to speed up both the forward and backward processes. 
More importantly, we improve the expressive power of diffusion models by employing both deterministic and stochastic mappings. Experiments on standard image generation datasets demonstrate the advantage of the proposed method over existing approaches. 
On the unconditional CIFAR10 dataset, for example, we achieve an FID of 2.01 and an Inception score of 9.96. Our method also demonstrates competitive performance on CelebA-HQ-256 dataset as it obtains an FID score of 7.11.
Code is available at $\href{https://github.com/MohsenZand/DiNof}{https://github.com/MohsenZand/DiNof}$.

Published in Transactions on Machine Learning Research (09/2024)
Diffusion Models with Deterministic Normalizing Flow Priors
Mohsen 2 Ali Etemad?, Michael Greenspan
Research Computing Center, University of Chicago
*Dpprtmentt of Electrical and Computer Engineering, and Ingenciity Labs Research Institute, Queen's University
zand@uchieags..dde ali. cle.eead@@uueenns.ccc michacel.rrensseeneeuueee
Reviewed on OpenReview: https: // opt/review.rwe... net/ forun? id- ACMNVwcR6U
Abstract
For faster sampling and higher sample quality, we propose DiNof (Diffusion with Normalizing flow priors), lechnique that makes use of normalizing flows and diffusion models. We use normalizing flows to parameterize the noisy data at any arbitrary step of the diffusion process and utilize it as the prior in the reverse diffusion process. More specif- ically, the forward noising process turns a data distribution into partially noisy data, which are subsequently transformed into a Gaussian distribution by a nonlinear process. The backward denoising procedure begins with a prior created by sampling from the Gaussian distribution and applying the invertible normalizing flow transformations detorministically. To generate the data distribution, the prior then undergoes the remaining diffusion stochas- tic denoising procedure. Through the reduction of the number of total diffusion steps, able to speed up both the forward and backward processes. More importantly, improve the expressive power of diffusion models by employing both deterministic and stochastic mappings. Experiments on standard image generation datasets demonstrate the advantage of the proposed method over cxisting approaches. On the unconditional CIFAR10 datasot, for example, we achieve an FID of 2.01 and an Inception score of 9.96. Our method also demonstrates competitive performance on CelebA-HQ-2555 dataset it obtains an FID score of 7.11. Code is available at https://githhbb.oo Madia/bboooooooooo/////////io
1 Introduction
Diffusion models (Sohl-Dickstein et al.) 2015: Dhariwal & Nichol 2021 Chen el al. 2020 Ho et al. 2020:
Kong et al. 2020 Song & Ermon 2020) are a promising new family of deep generative model that have recently shown remarkable success on static visual data, even beating GANs (Generative Adversarial Net- works) Goodfellow et al. 2014) in synthesizing high-quality images and audio. In a diffusson model, noise is gradually added to the data samples using a predetermined stochastic forward process, converting them into simple random variables. This procedure is reversed using a separate backward process that progressively removes the noise from the data and restores the original data distributions. In particular, the deep neural network is trained to approximate the reverse dilfusion process by predicting the gradient of the data density.
Wehenkel & Louppe 2021 define a Gaussian distribution as the prior noise, and non-parametric diffusion method is developed to procedurally convert the signal into the prior noise. The traditional Gaussian prior is simple to apply, but since the forward process fixed, the data itsclf has no impact on the noise that is introduced. As a result, the learned network may not model certain intricate but significant. characteristics within the data distribution. Particularly, employing purely stochastic prior noise in modeling complex data distributions may not fully leverage the information and completely encompass all data details in the diffusion models. *This work was partially done when Mohsen Zand was Postdoctoral Fellow Queen's University.
Existing diffusion models (Yang et al. 2022 Zhang & Chen 2021

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 1: Uncurated samples generated by DiNof on CelebA-HQQ250 (left) and CIFAR-10 (middle) datasets. On the right figure, sample quality in terms of FID is shown versus Iuffusion/samppll steps for different diffusion-based generative models. As opposed to several other methods, DiNof can speed up the process while improving the sample quality.
Another issue that the sampling procedure in most current methods involves hundreds or thousands of steps (time-dicrrcizzation stages) (Salimans & Ho. 2022 Ramachandran al. 2017 Zhang & Chen. 2021) This is due the fact that the noise in the forward process must. be added to the data at slow enough rate to enable successful reversal of the forward process for the reverse diffusson to produce high-quality samples. This, however, slows down training and sampling because it requires sufficiently long trajectories (the paths between the data space and the latent space), resulting in substantially slower sampling rate
than, for example, GANs or VAEs (Lu et al. 2022 Zhang & Chen 2022 single-step at inference.
Salimans & Ho 2022
which are
In this work, we leverage the use of flow-based models, learn noise priors deterministically from the data itself, which is then applied to improve the effectiveness of diffusion-based modelling. We propose the use of deterministic prior as alternative to the completely random noise of conventional diffusion models. We specifically develop novel diffusion model where the data and latent spaces are connected by the nonlinear invertible maps from normalizing flows. Our model thus preserves all the benefits of the original diffusion model formulation, while employing both deterministic and stochastic trajectories in the mappings between the latent and data spaces. Our hypothesis is that data-informed latent feature representations and using both stochastic and deterministic processes can improve the representational quality and sample fidelity of generative models. We can generate samples using fewer sampling steps while attaining a sample quality that is superior to cxisting models (sce Figure D. Moreover, our model can be used in both conditional and unconditional setlings, which can enable various image and video generation lasks.
Our contributions can be summarized as follows:
(1) We propose the use of a data-dependent, deterministic prior as an alternative to the random noise used in standard diffusion models, to enable modeling complex distributions more accurately, with a smaller number of sampling steps. Our new generative model leverages the strengths of both diffusion models and normalizing flows to improve both accuracy and efficiency in the mappings between the latent and data spaces.
(2) We evaluate our approach on CIFAR-10 and CelcbA-HQ-256 datascts, which are the most commonly used datasets in this problem space. We achieve competitive results in the image generation task, yielding 2.01 and 7.11 FID scores on the CIFAR-10 and Celeb datasets, respectively.
(3) To allow reproducibility and contribute to the area, we release our code at:
https//gtthlbbcom//Moooon//////d
2
Related Work
There has been previous research looking towards creating a more informative prior distribution for deep generative models. For instance, hand-crafted priors (Nalismick & Smyth 2016 Tomczak Welling 2018
vector quantization (Razavi et al.] 2019), and data-dependent priors [Li et al. 2020 Lee et al. 2021 have
2

---

Published in Transactions on Machine Learning Research (09/2024)
been proposed in the literature. As priors for variational autoencoders (VAE), normalizing flows and hi- erarchical distributions (Maaløe et al. 2019 Child
2020 Vahdat & Kautz 2020 Rezende & Mohamed 2015: Kingma et al.] 2016) have been employed in particular. Prior distributions have also been defined Takahashi et al.) 2019 Mittal et al. (2021) parame-
implicitly (Bauer & Mnih, 2019 Aneja et al. 2020
terized the discrete domain in the continuous latent space for training diflusion models on symbolic music data. They conducted two independent stages of training for a VAE and the denoising diffusion model. Wehenkel & Louppe (2021) proposed an end-to-end training method that modeled the prior distribution of
the latent variables of VAEs using Denoisng Diffusion Probabilistic Models (DDPMs). They demonstrated
that the diffusion prior model outperformed the Gaussian priors of traditional VAFs and was competitive with normalized flow-based priors. They also showed how hierarchical VAEs might profit from the improved capability of diffusion priors. Sinha el al. (2021) combined contrastive learning with diffusion models in the latent space of VAEs for controllable generation.
More recently, Lee et al. (2021) proposed PriorGrad to improve conditional denoising diffusion models with data-dependent adaptive priors for speech synthesis. They calculated the statistics from conditional data and utilized them as the Gaussian prior's mean and variance. Vahdat et al. (2021) proposed the Latent Scoee-based Generative Model (LSGM), a VAE with a score-based generative model (SGM) prior. They used the SGM after mapping the input data to latent space. The distribution over the data set embeddings was then modeled by the SGM. New data synthesis was accomplished by creating embeddings by drawing data from a base distribution, iteratively denoising the data, and then converting the embedding to data space via decoder.
Denoising Diffusion Implicit Model (DDIM) proposed by Song et al. (2020a) was basically fast sampling algorithm for DDPMs. It creates a new implicit model with the same marginal noise distributions DDPMs while deterministically mapping noise to images. Specifically, an alternative non- Mar-ooioo noising process is developed that has the same forward marginals as the DDPM but enables the generation of various reverse samplers by modifying the variance of the reverse noisc. An implicit latent space is therefore created from the deterministic sampling process. This is equivalent to integrating an ordinary differential equation (ODE) in the forward direction, followed by obtaining the latents in the backward process to generate images. Salimans & Ho| (2022) proposed to roduce the number of sample steps in progressive distillation model. The knowledge of trained teacher model, represented by deterministic DDIM, is distilled into student model with the same architecture but with progressively halved sampling steps, thereby improving efficiency.
Slow generating speed continues to be a significant disadvantage of diffusion models, despite the outstanding performance and numerous variations. Different strategies have been investigated to overcome the efficiency issue. Rombach et al.] (202)) used pre-trained autoencoders to train a diffusion model in a low-dimensional representational space. The latent space learned by an autoencoder was employed for both the forward and backward processes. Deterministic forward and reverse sampling strategics were suggested by DDIM (Song et al.
2020a to increase generation speed.
Similar to our approach, Zhang & Chen 2021 connected normalizing flows and diffusion probabilistic models, and presented diffusion normalizing flow (DillFlow) based on stochastic differential equations (SDEs). The reverse and forward processes were made trainable and stochastic by expanding both diffusion models and normalizing flows. They extended the normalizing flow approach by progressively introducing noise to the sampling trajectories to make them stochastic. The diffusion model was also expanded by making the forward process trainable. By minimizing the difference between the forward and the backward processes in terms of the Lullback-Liiber (KL) divergence of the induced probability measures, the forward and backward diffusion processes were trained simultancously. Kim et al. (2022) also combined normalizing flow and a diffusion process. They proposed an Implicit. Nonlinear Diffusion Model (INDM), which implicitly constructed a nonlinear diffusion on the data space by leveraging a linear diffusson on the latent space through a flow network. The linear diffusion was expanded to trainable noulinear diffusson by combining an invertible flow transformation and a diffusion model. Recently, Zhang et al. (2023) developed UnifiedDifFloow a unified theoretic framework for score-based diffusion models and GANs. It enabled flexible trade-off between high sample quality and fast sampling speed by utilizing neural networks to approximate the SDE dynamics.
3

---

Published in Transactions on Machine Learning Research (09/2024)
Truncated diffusion probabilistic modeling (TDPM) (Zheng et al. 2023). an adversarial auto-encoder em- powered by both the diffusion process and learnable implicit prior, is another method in this vein. Similar to LSGM, TDPM utilized variational autoencoder when transitioning from data to latent space. Specifically, TDPM is most closely related to an adversarial auto-encoder (AAE) with a fixed encoder and a learnable decoder, which uses a truncated diffusion and learnable implicit prior. therefore a diffusion--based AAE that emphasizes shortening the diffusion trajectory through learning an implicit generative distribution.
Another related approach that addresses the issue of sampling from diffusion models to produce high-quality
images is the diffusion probabilistic model based on optimal transport (DPM-OT) (Li et al. 2023 DPM- OT conceptualizes the process of inverse diffusion as an optimal transport (OT) problem, focusing on the transition between latent representations at different stages.
In contrast to existing techniques, our method uses SDEs and ODEs to map between data space and latent space utilizing both lincar stochastic and nonlincar deterministic trajectories. We concentrate on nonlin- earizing the diffusion process by using nonlinear trainable delermónislic processes via nonlinear invertâble flow mapping.
3 Method
Although diffusion models have unique advantages over other generative models, including stable and scalable training, insensitivity to hyperparameters, and mode-collapsing resilience, producing high-fidelity samples from diffusion models involves fine discretization sampling process with sufficiently long stochastic tra- jectories (Sng et al. 2020b Vahdat et al. 2021 Zhang & Chen 2021 Zhang et al. 2022). Several works investigated this and improved the efficiency. However, their improved runtime comes at the expense of poorer performance compared to the seminal work by SSng et al. (020b
This motivates us to develop a new approach that simultaneously improves the sample quality and sampling time. We hence propose to integrate nonlinear deterministic trajectories in the mapping between the data and latent spaces. The deterministie trajectories are learned by using normalizing flows. In our diffuuiion/smmplig /sampling steps, therefore, both stochastic and deterministic trajectories employed. In the following subsections, we provide preliminary remarks on diffusion models and normalizing flows before introducing our approach.
3.1 Background
3.1.1
Diffusion Models
Diffusion models are latent variable models that represent data x(0) through underlying series of latent variables {x()) The key concept is to gradually destroy the structure of the data x(0) by applying a diffusion process (i.e,, adding noise) to it over the course of T time steps. The incremental posterior of the diffusion process generates x() through a stochastic denoising procedure (Yang ct al.] 2022: Rasul et al.
2021
Voleti et al. 2022 Ho et al. 2022 Croitoru et al. 2022 Ho et al. 2020 Song et al. 2020a).
The diffusion process (also known as the forward process) not trainable and fixed a Markov chain that gradually adds Gaussian noise to the signal. For a continuous time variable t E {0,,"), the forward diffusion process {T is defined by Itô SDE as:
dx +())
(1)
where f(.,t): Rd + Rdd and g(.) R denote a drift term and diffusion coefficient of x(t), respectively. Also, w denotes the standard Wiener process (known as Brownian motion). To obtain a diffusion process as a solution for this SDE, the drift coefficient should be chosen so that gradually diffuses the data x(0), while the diffusion cocfficient regulates the amount of added Gaussian noise. The forward process transforms x(0) ~ Po into simple Gaussian x(T) ~ Pr so that at the end of the diffusion process, pr is an unstructured prior distríbution that contains information of po, where (x(x) denotes the probability density of x x())-
4

---

Published in Transactions on Machine Learning Research (09/2024)
It is shown by Song et al. (2020b) that the SDE in Eq. 1 can be converted to generative model by starting from samples of (T') pr and reversing the process as reverse-time SDE given by:
dx = t) ( log g(t)dw,
(2)
where w denotes a standard Wiener process when the time reversed from T to 0 and df denotes an infinitesimal negative time step. We can formulate the reverse diffusion process from Eq. | and simulate it to sample from Po after determining the Vx Vxlog score for each marginal distribution for all We can thereby restore the data by eliminating the drift that caused the data destruction.
A time-dependent score- model s0(x, 6) can be trained to estimate Vx (x) at time t ~~ U[0, by optimizing the following objective:
= arg min E, EA - -occ Por (x)))
6
where A: [0 + -x denotes a weighting function, x(0) Pz),,, x(t) (x)) and Por denotes the transition from x()) to xt).. By using score matching with enough data and model capabilities, the optimum solution 5g. (x, t) can be achieved for nearly all x and t. It is hence equivalent to Vx log (x()..
To solve Eq. 3 the transition kernel Por (x)())xx() must be known. For an affine drift coefficient f. it is usually Gaussian distribution, whose mean and variance are known in closed-forms. Once time-dependent score- based model so has been trained, it can be utilized to create the reverse-time SDE. It can then be simulated using numerical methods to generate samples from po- Any numerical method applied to the SDE specified in Eq. can be used to carry out the sampling. Song et al. (2020b) introduced some new sampling techniques, the Predictor-Corroctor (PC) sampler being one of the best at generating high-quality samples. In PC, at each lime step, the numerical SDE solver is used as a predictor to give an estimate the sample at the next time step. Then, score-based technique, such as the annealed Langevin dynamics (Song & Ermon 2019) is used as a corrector to correct the marginal distribution of the estimated sample. The annealed Langevin dynamics algorithm (Song & Ermon 2019) begins with white noise and runs x; x,-1+ 2 log p(x)+ /Tw,, certain number of iterations, where y controls the magnitude of the update in the direction of the score Vx log p(x).
The reverse-time SDE can also be solved using probability flow, a different numerical approach, thanks to score-based models (Song et al., 2020b). corresponding deterministic process with trajectories that have the same marginal probability densities {pr(x))<"o as the SDEs exists for every diffusion process. This process is deterministic and fulfills an ordinary differential equation (ODE) (Song et al. 2020b):
dx [[(x - (/)G(()((( Vx lo (x)]dr.
(4)
This process is called probability flow ODE. Once the scores are known, one can determine it from the SDE,
3.1.2 Normalizing Flows
In normalizing flows, random variable with known (usually Normal) distribution is transformed via series of differentiable invertible mappings (Abdelhamed et al., 2019 Kobyzev et al. 2020 Zhang & Chen 2021 Zand et al. 2023). Tet Z E D be random variable with the probability density function pz " - R being a known and tractable function and Y = gZ). The probability density function of the random variable Y can then be calculated using the change variables formula as shown below:
0f pz(s(v)) det af())
0g
pr(y) = pz(l(w)) del Fy
1-1
(5)
0g where उフ() denotes the Jacobian of f, and f is the inverse of g.
In generative models, the aforementioned function g (a generator) 'pushes forward' the initial base density Pz, often known as the 'noise', to a more complicated density. This the generative direction, which a data point y generated by sampling & from the base distribution and applying the generator as y = g(2). In order to normalize complex data distribution, the inverse function f moves (or 'flows') in the opposite
5

---

Published in Transactions on Machine Learning Research (09/202))
x(0)
x(0)
x(O)
x(O)
x(Tg-21 21
x(T
x 3
x
Score network
NF network
x
xIT
xIT
xlT
Backward/Sampling
Doward/Difffssing
x(T
m
x(T
x
+++
x(T)
X/TI
x(T)
Diffusion Models
Normalizing Flows DiftFlow (2hang & Chen, 2021) DiNof (ours)
Figure 2: Architectural comparison of Diffusion, Normalizing Flows, DiffFlow ZZhang & Chen 2021 and the proposed DiNof models. DiffFlow uses stochastic and trainable processes for both the forward and the backward processes, whereas DiNof utilizes a deterministic trainable process only at the final steps of the forward process. The backward process initiates with deterministic process and turns to a stochastic process generates images. We use Tm to denote an arbitrary intermediate latent variable between data space and latent space.
way, from the complex distribution to the simpler, more regular or 'normal' form of pz. Given that is 'normalizing' the data distribution', this viewpoint the source of the term 'normalizing flows'.
It can be challenging to build arbitrarily complex nonlinear invertible functions (bijections) such that the determinant of their Jacobian can be caleulated. To address this, one strategy is to use their composition, since the composition of invertible functions is itself invertible. Let 91,...,9ht 9MM be a collection of M bijective functions, and let = 9Ms o 9M-1 ..... 91 represent the composition the functions. The function g can then be demonstrated to be bijective as well, which its inverse given as f f1 o.... JM o SM- The latent variables are then given as:
x, = f.
(6)
X,-1 f. (x,, fx,(
where M denote the trajectories between the data space and the latent space.
3.2 Proposed Method
A schematic illustration of the proposod mothod comparison other generative models shown Figure _ In diffusson models, the forward process fixed while the backward process is trainable. Yet, they are both stochastic. Both the forward and the backward processes of normalizing flow deterministic. They combine into single process since they are the inverse of one another. In DiffFlow (Zhang & Chen 2021), both the forward and the backward processes are stochastic and trainable. Our method however employs both stochastic and deterministic trajectories that follow each other in both directions. This is more effective due to the possible reduction of the diffusion /sampling steps.
diffusion/sampling
More specifically, we aim to improve the effectiveness of diffusion- based modelling by representing data x(0) via set of latent variables x(t) between the data distribution and data-dependent and deterministie prior

---

Published in Transactions on Machine Learning Research (09/2024)
X( D
Linear Fixed Stochastic Forward Path
Linear Stochastic Backward Path
x(0)
x (m)
(m)
x(0)
Nonlinear
Nonlinear
Trainable
Deterministic
Deterministic
Backward
Forward Path
Path
Figure 3: An overview of DiNof. It employs both linear stochastic and nonlinear deterministic trajectories in the mapping between data space and latent space using SDEs and ODEs. It hence utilizes normalizing flows to nonlinearize the diffusion models. Glow (Kingma & Dhariwal 2018 architecture is used as the normalizing flow model.
distribution. We use a total number of N noise scales and define x( x,, where {x+ denotes the Markov chain. An SDE and an ODE both are used to model the forward diffusion process. We employ an Itô stochastic SDE as follows:
dx = f()xxdt + g(t)dw,
(7)
where é is continuous time variable uniformly sampled over [0, (m... Theoretically Tm. can be any number between 1 and T. implying a potential for reducing the diffusion steps. Furthermore, we model latent variables (x()))<e.., in the forward process using Fq. 7 Here, we can use one of the original diffusion models, such as VESDE (variance exploding SDE) or VPSDE (variance preserving SDE) (Song et al. 2020b). These models are linear, where f(x, ft)x is a function of x(f), and g function of
In VESDE, f()) and s(t) ddt2/dt, where o²(t) denotes the variance of latent variable x x(t)- Com- parably, [(1) = - 1/) and g()) = /B(I) VPSDE, where {(x)= = 3,, and (3, = N3. We can also employ the discretized versions of the VESDE and VPSDE known as SMLD and DDPM noise per- turbations (Song et al. 2020b). Using these conventional linear SDEs in the forward diffusion process, we transform the data x(0)= Pu diffused distribution pr.. Hence, connect the data space and the latent variables (x(1))eeT.. through stochastic trajectories.
We propose exploiting the nonlinearity in the diffusion process by applying a nonlinear ODE on the rest of trajectories (é.e,, (x(t))>>T..) In contrast to a few prior works that use nonlincarity in the diffusion models (Vahdat et al. 2021 Zhang & Chen 2021 Kim et al. 2022), we follow the existing linear process with a subsoquent nonlinear process. This is shown Figure where the diffusion process nonlinearized by employing nonlinear trainable deterministic processes. Intuitively and empirically, utilizing both linear and nonlinear processes can boost the sample quality. As demonstrated in our experiments, executing efficient nonlinear process after an existing linear process can reduce the mumber of sampling steps and accelerate sampling. We choose normalizing flows as the means to nonlinearize the diffusion models, as they learn the nonlinearity by invertible flow mapping. Normalizing flows are in this way used to complete the remaining steps of the diffusion process in a single phase, which improves efficiency.
In our normalizing flow network, we consider a bijective map between pr.. and 2, latent variable with simple tractable density such Gaussian distribution as po(2) (z; I) The log-likelihood of

---

Published in Transactions on Machine Learning Research (09/2024)
x(Tm) is then defined as:
logpo (a) = log po()) + log (dt/d//))) log Σ log (dh.
(8)
where [b: are intermediate representations generated by the layers of neural network, ho = x(Tm) and hM 2, We train this model by minimizing the negative log-likelihood. The overall objective which includes training our SDE and our ODE is a joint training objective that merges the ODE objective with the diffusion model's score matching objective Le., Eqs. 3 and
As noted earlier, data and the latent variables are coupled through both stochastic and deterministic tra- jectories in end-to-end network. We therefore employ two different forms of trajectories for the mapping between data and latent spaces using SDEs and ODEs. Stochastic trajectories are utilized between T' latent variables, while deterministic trajectorics are employed for M latent variables in the flow process. Although Tm+ might be greater than the N steps of a standard diffusion model, employing an efficient flow network which generates samples at a single step will nevertheless, speed up the process (Song et al. 2020b).
The typical strategy in current diffusion models is to restore the original distribution by learning pro- gressively reverse the diffusion process, step by step, from T to O. In our approach, however, reconstruct x(Tm.) using the backward process in the flow network via single path. Furthermore, by reconstructing x(Tm), the Gaussian noise is deterministically mapped to a partially noisy sample pr... New samples are generated in the backward process by simulating remaining stochastic trajectories from t = Tm to t = 0 by the reverse-time SDE.
In contrast to generic SDEs, we have extra information close to the data distribution that can be leveraged to improve the sample quality. Specifically, Pr... which sampled by the flow network used as an informative prior for the reverse-time diffusion. To incorporate the reverse-time SDE for sampling, we can use any general SDE solver. In experiments, we choose stochastic samplers such as Prcdictor-Correcto (PC) to incorporate stochaslicily in the process, which has been shown to improve results (Song et. al. 2020b). Another notable advantage of this approach is its ability to semantically modify images by changing the value of Tm- It can interpolate between deterministic and stochastic processos. In our experiments, wo demonstrate the impact of the value Tm on sample quality.
4 Experiments
We conduct a systematic evaluation to compare the performance of our method with competing methods in terms of sample quality on image generation.
4.1 Protocols and Datasets
We show quantitative comparisons for unconditional image generation on CIFAR-10 Krizhevsky et al.
2009) and CelebA-HQQ-256 (Karras et al. 2017). We perform experiments on these two challenging dalasets
following the conventional experimental setup in the field (such as Kim et al.
(002)); Salimans & Ho (022)
Song et al. (2020a)). ). We follow the experimental design of Ho et al. (2020);
Song et al. (002b)). using the
Inception Score (IS) [Salimans et al. 2016) and Frechet Inception Distance
(FID) Heusel et al. 2017 for
comparison across models.
We use different SDEs such VESDE, VPSDE, and sub- VPSDE to show our consistency with the existing approaches. The NCSN++ architecture is used as our VESDE model whereas DDPM++ architecture is utilized for VPSDE and sub- models. PC samplers with one corrector step per noise scale are also used to generate the samples. As our normalizing flow model, we use the multiscale architecture Glow (Kingma & Dhariwal 2018) with the number of levels L = 3 and the number of steps of each level K = 16. We also set the number hidden channels to 256.
To ensure that the amount of neural network evaluations required during sampling consistent with prior work (Ho et al.) 2020 Jing et. al. 2022 Vahdat al. 2021 Song al. 2020b), set T 1000 and T

---

Published in Transactions on Machine Learning Research (09/2024)
Table Effect varying (mm 11/10 on the CIFAR-10 dataset, where =
12
Model
Tm
samp. steps
IS
FID
NCSN ++ cont. (Song
2020b
1000
8.91
4.29
0.1
100
5.29
46.60
0.2
200
6.11
34.63
0.3
300
7.50
21.32
0.4
400
8.82
10.76
0.5
500
9.41
3.94
DiNof
0.6
600
9.23
3.16
0.7
700
9.13
3.29
0.8
800
8.99
3.95
0.9
900
9.21
3.76
1.0
1000
8.87
4.18
Table 2: Generative performance on CIFAR-10 dataset
Class
SDE
Type
Method
IS
FID
StyleCAN2-ADA Karras et al.
2020
9.83
2.92
SNGAN + DGflow
Ansari
2020
9.62
GAN
TransGAN Jiang et al. 2021
9.26
StyleFormer Park & Kim
2022
2.82
StyleSAN-XL Takida et a..
2023
1.36
NVAR [Vahdat & Kautz 2020
23.5
VAE
DCVAE Parmar
2021
17.9
CR-NVAE (iinha & Dieng
2021
2.51
Glow (Kingma & Dhariwa
2018
46.90
ResFlow (Chen et al. 2019)
46.37
Flow
Flow+ (oo el al. 2019
46.4
DenseFlowe 1ow-74-10 GGrcié et al.
2021
34.9
NCSN (Song & Ermon (2019)
8,87
25.32
NOSN (Sng & Ermon 2020)
8.40
10.87
NCSN + cont. (deep. VE)
Song et al.
2020b
9.89
2.20
Linear
DDPM Ho al. 2020
9.46
3.17
DDIM (Song et 2020a
4.04
Distillation (Salimans
2022
2.57
Subspace Dift. (NSCN+
deep)
Jing et al.
2022
9.94
2.17
Diffusion
SBP
SB-FBSDE (Chen et al.
2021)
3.01
OT
DPM-OT (Li al. 2023
2.92
LSGM (FID) (ahddtt et al.
2021
2.10
Nonlinear
VAE-based
LSGM (NLL) Vahdat al.
2021
6.89
TDPM (Treune=99) Zheng et al.
2023
2.83
DiffFlow (Zhang Chen
2021
13.43
INDM (FID) (Kim et al.
2022
2.28
Flow-based
INDM (NLL)
Kim
2022
4.79
DiNof (Ours)
9.96
2.01
for discrete and continuous diflusion processes, respectively. The number of noise scales N is however set to 1000 for both cases. Additionally, the number of conditional Langevin steps is set to 1. The Langevin signal-to-noise ratio for CIFAR-I0 and CelebA-HQ-256 are fixed at 0.16 and 0.17, respectively. The default settings are fixed for all other hyperparameters based on the optimal parameters determined in (Song al.
2020bb Jing al. 2022h. All details available in the source code release.

---

Published in Transactions on Machine Learning Research (09/2024)
Table 4: Generative performance and sampling time on CIFAR-10
Table 3: Generative performance on CelebA-HQ-226 dataset
Model
IS
FID
Time
DDPM++
9.64
2.78
43s
TDPM (Trrane=D9, DDPM++)
9.62
2.83
37s
Method
FID 1
DiNof (Glow, DDPM-
9.65
2.51
24s
Glow (Kingma & Dhariwal
2018
68.93
DDPM++ cont. (VP)
9.58
2.55
458
NVAE
Vahdat & Kautz
2020
29.76
DiNof (Glow, DDPM+4 cont. (VP))
9.75
2.40
258
DDIM
Song al.
2020a
25.60
DDPM++ cout. (sub-VP)
9.56
2.61
448
SDE
Song et al.
2020b)
7.23
DiNof (Glow, DDPM cont. (sub-
9.73
2.43
24s
D2C
Sinha et
2021
18.74
NCSN++
9.73
2.45
97s
LSGM
Vahdat et al.
2021
7.22
DiNof (Glow, NCSN++)
9.85
2.25
56s
TDPM
Zheng et al.
2023
8.38
NCSN++ cont. (VE)
9.83
2.38
97s
RDM
Teng et al.
2023
3.15
DiNof (Glow, NCSN++ cont. (VE))
9.87
2.12
56s
DiNof (Ours)
7.11
NCSN++ cont. (deep, VE)
9.89
2.20
150s
DiNof (Glow, NCSN++ cont. (deep, VE)) 9.96
2.01
90s
Tm 0.1
T 0.2
= 0.3
T 0.4
T,,=Oa 0.5
Tm 0.6
T 0.7
T 0.8
T = 0.9
Tm = 1.0
Figure 4: CIFAR-10 samples with different Tm thresholds.
4.2
Results
4.2.1 Model Parameters
We first optimize for the CIFARIO sample quality, and then we apply the resulting parameters to the other dataset. To find the optimal value for Tm, we investigate the results for a variety of thresholds. We select NCSN++ cont. (Song et al., 2020b), which is NCSN+ model conditioned continuous time variables as the baseline model. We calculate FIDs for various Tm values with T T/10 /100 increments on the models trained for 500K training iterations with batch size of 32, where T = 1. Depending on the Tm value, different number of sampling steps is used in our model. Note that the number of sampling steps are reduced except when Tm = T. In this case, normalizing flows provide an initial prior as an alternative to the standard Gaussian prior.
As reported in Table | the best IS and FID are obtained when Tm 0.5 and Tmm 0.6, respectively. Also, our method with Tn 0.5 and 500 fewer sampling steps, outperforms the baseline model. It achieves 0.35 improvement in terms of FID over the baseline method by obtaining an FID 3.94. Improvements be seen for all Tm 2 0.5. For smaller T., values, however, our method suffers significant. degradation. This is due to the high and imbalanced stochasticity at smaller T'm values. This is shown in Figure g where unrecognizable images are generated with a high stochasticity for Tn < 0.3. Smaller Tm makes the backward process simple and fast. but challenging to reconstruct the data. By additional noise, however, the high-quality images are successfully reconstructed, although with more sampling steps. Nonetheless, the capacity to explicitly trade-off bet between accuracy and efficiency is still crucial feature. For instance, the number of function evaluations is decreased by nearly 50% while maintaining the visual quality of samples by using smaller threshold, such as Tm = 0.5 that balances sample quality and efficiency (i.e., the number of sampling steps). We fix Tm = 0.6 for the rest of the experiments as it results in the best FID score of 3.16.
4.2.2 Unconditional Color Image Generation
We compare our method with prominent diffusion-based models and nonlinear diffusions, providing a repre- sentative evaluation within the current state-of-the-art landscape. There have been few prior works that
10

---

Published in Transactions on Machine Learning Research (09/2024)
have leveraged diffusion models with nonlinear diffusions. Specifically, the examples found in the literature
are:
LSGGM (Vahdat et al., 2021 which implements a latent diffusion using VAE; DiffFlow (Zhang & Chen
2021
which uses a flow network to nonlinearize the drift term; SB-FBSDE (Chen et al. 2021 De Bortoli
ict al.
2021 which reformulates the diffusion model into a Schrodinger Bridge Problem (SBP); DPM-OT LLi
let al.
2023 which treats inverse diffusson as an optimal transport (OT) problem between latents various
stages; TDPM (Zheng et al.) 2023) that focuses on reducing the diffusion trajectory by learning an implicit
generative distribution using AAE, and; INDM (Kim et al. 2022) which uses flow network to implicitly construct a nonlinear diffusion on the data space by leveraging a linear diffusion on the latent space. We show that, compared to these methods, our flexible approach provides better generative modeling perfor- mance. We demonstrate this on the CIFAR-10 image dataset, which compared to CelebA-HQQ226 is far more diversified and multimodal.
As is done previous research {Song et al. 2020b Jing et al. 2022 the best training checkpoint with the smallest FID is utilized to report the results on CIFAR-10 dataset. One checkpoint saved every 50k iterations for our models, which have been trained for 1M iterations. The batch size is also fixed to 128.
Table 2 summarizes the sample quality results on the CIFAR-10 dataset for 50K images. Our method with continuous NCSN++ (deep) obtains the best FID and IS of 2.01 and 9.96, respectively. The considerable im- provement over other nonlinear models such as DiffFlow, SB-FBSDE, INDM, LSGM, and TDPM shows the potential of using both SDEs and ODEs. It is worth noting that although certain methods like DPM-OT Li itt al. (2023) might require fewer steps in the reverse process, our method still maintains a competitive edge in terms of sample fidelity. More importantly, our method maintains full compatibility with the underlying diffusion models, and hence, retaining all their capabilitics. Another key point is that our approach, in comparison to similar methods that integrate normalizing flows with diffusion models, like DiffFlow (Zhang & Chen 2021), delivers notably better performance. The strength of our method lies in optimizing computa- tional efficiency without compromising sample quality. With an FID score of 2.01, as opposed to DiffFlow's 13.43, our approach demonstrates that it can both accolerate the process and improve sample quality. Our method also consistently improves over the baseline methods. For example, achieves a 0.19 improvement in the FID score compared to NCSN++ cont. (deep, VE) with the same diffusion architecture. Furthermore, our method not only proserves the flexibility of existing SDEs but also boosts their effectiveness.
We evaluate the applicability of our method to the high- resolution dataset of H2-25. We trained on this dataset for 0.5M iterations, and the most recent training checkpoint is used to derive the results. We use a batch size of for training and a batch size of 64 for sampling. To save on computation, we use the optimal Tm value of 0.6, which is obtained on the CIFAR-10 dataset. We also utilize the continuous NCSN+ with PC samplers. As shown in Table 3. DiNof achieves competitive performance in terms of FID on the CelebA-HHQ-256 dataset. It specifically obtains a competitive FID of 7.11 which outperforms LSGM, another nonlinear model by a considerable margin of 0.11 FID. The performance gain is mainly contributed to the integrated deterministic nonlinear priors as our diffusion architecture is similar to the one used in SDE and LSGM.
Table 5: Intermediate results for different it- eration numbers on the CIFAR-10 datasel.
Model
Iteration
IS t
FID 4
50k
8.62
6.05
100k
8.97
3.71
150k
9.34
2.99
D
200k
9.50
2.72
250k
9.57
2.70
300k
9.60
2.69
50k
8.48
6.13
100k
8.96
3.83
150k
9.22
3.05
DDPM+
200k
9.43
2.79
cont. (VP)
250k
9.58
2.66
300k
9.71
2.69
4.2.3 Sampling Time
50k
8.32
7.55
100k
8.75
4.91
We evaluate DiNof in terms of sampling time the CIFAR- 10 datasct. We specifically mcasure the improved runtime in comparison to the original SDEs (VESDE, VPSDE, and sub- VPSDE) with PC samplers (Song et al., 2020b) on an NVIDIA A100 GPU. The sampler is discretized at 1000 time steps for all SDEs. We however discretize it at 600 steps in our models. Thus, we employ considerably smaller number
150k
9.00
3.85
DDPM+
200k
9.15
3.35
cont. (sub-
250k
9.31
3.11
300k
9.37
2.94
11

---

Published in Transactions on Machine Learning Research (09/2024)
100k 150k
250k
300k iterations
350k iterations
iterations
450k
50Ok iterations
Figure 5: Visual samples for various ileration numbers during training on the CelebA-HQ--20 dataset.
of diffusion diffusion/samppling /samllng steps. As we employ a Glow architecture, our models include ~ 7M more parameters than the standard SDE models. In contrast to the SDE models which are trained for 1.3M iterations, we train our models for 1M iterations to suppress overfitting. We follow the same strategy as (Song et al. 2020b), and report the results on the best training checkpoint with the smallest FID.
As shown in Table 4 our method consistently reduces sampling time and improves sample quality. For instance, it takes 24s to generate an image sample, while yielding an FID score of 2.51 on the DDPM-+ model. It however takes 43s using the original DDPM++ architecture which achieves FID = 2.78. Our method is hence ~ 1.7x faster than original SDEs. It also performs better in terms of FID and IS. For instance, it improves over SDEs by 1.1x FID on average.
4.2.4 Intermediate Results
We save one checkpoint every 50k iterations for our models and report the results on the best training checkpoint with the smallest FID. It however worthwhile considering the intermediate results to better understand the entire training and inference procedures. In Table we show IS and FID for DDPM++ DDPM++ cont. (VP), and DDPM+ cont. (sub- VP) models trained for various iteration numbers on the CIFAR-10 dataset. As can be observed, DDPM+ and DDPM++ cont. (VP) models improve more quickly than DDPM++ cont. (Sub-VP).
Visual samples during training are also illustrated in Figure 5 for the CelebA-HQ-28 dataset, where im- provements over the course of training are obvious. Further details are updated and modified as training progresses.
4.2.5 Qualitative Results
We visualize qualitative results for CelebA-HQ-256 and CIFAR-10 in Figure | We show that as opposed to several other methods, DiNof can speed up the process while improving the sample quality. Other methods that shorten the sampling process like (Song et al. 2020a. Zhang & Chen 2021 frequently compromise the sample quality.
Random samples from our best models on the CIFAR-10 and CelebA-HQQ datasets are further depicted in Figure and Figure 7 respectively. They demonstrate the robustness and reliability of our approach for generating realistic images. Our method creates diverse samples from various age and ethnicity groups on CelebA-HQ-2550 logether with range of head postures and face expressions. DiNof also produces sharp and high-quality images with density details on the challenging multimodal CIFAR-10 dataset.
5 Conclusion
We propose DiNof, which improves sample quality while also reducing runtime. Our model breaks previous records for the inception score and FID for unconditional generation on both CIFAR-10 10 and CelebA-HQ- 256 datascts. As compared to the previous best diffusion-basod generative models, is surprising that we are able to reduce the sampling time while improving the sample quality. Unlike many other methods, our approach also maintains full compatibility with the underlying diffusion models and retains all their
12

---

Published in Transactions on Machine Learning Research (09/2024)
Figure 6: Uncurated CIFAR-10 generated samples. Figure 7: Uncurated CelebA-HQQ226 generated samples.
features. It should also be noted that even though incremental experiments used to determine the optimal value of the single model hyperparameter (Tm), we only needed to run 10 increments (on CIFAR-10) to beat the other SOTA methods and show how useful DiNof is in real-world situations. We did not conduct this experiment again to acquire our results on the CelebA-HQ-256 dataset, further demonstrating the robustness and applicability of our technique.
A variety of image and video generation tasks may be made possible by our model's ability to applied both conditional and unconditional scenarios. It beneficial investigate its effectiveness for several other tasks including conditional image generation, interpolations, colorization, and inpainting.
Statement of Broader Impact
Our work enhances the efficacy of generative models image generation, offering potential applications creative tools, media accessibility, and computer vision research. However, these advancements carry risks, including the misuse of AI for creating misleading harmful content, such deepfakes, and amplifying biases in training data. We advocate for responsible Al use, adhering to ethical guidelines and transparency in deployment to mitigate these concerns.
References
AbdeIrahman Abdelhamed, Marcus A Brubaker, and Michael S Brown. Noise flow: Noise modeling with conditional normalizing flows. In Proceedings of the IEEE/CVE International Conference on Compuler Vision, pp. 3165-3173, 2019.
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. Nep-vae: Variational autoencoders with noise contrastive priors. 2020.
Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discriminator gradient flow. In International Conference on Learning Representations, 2020.
Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In The 22nd International Conference Artificial Intelligence and Stalistics, pp. 66 36-75. PMI.R, 2019.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Leorning Represenlations, 2020.
Ricky TQ Chen, Jens Behrmann, David K Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
Tianrong Chen, Guan-Horng Liu, and Evangelos A Theodorou. Likelihood training schrödinger bridge using boward-backward sdes theory. arXiv preprint arXiv:2110.1129) 2021.
Rewon Child. Very deep vacs generalize autoregressive models and can outperform them on images. arXiv preprial ar Xiv:2000 1. 10650, 2020.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. arXie preprinl arXiv:2209.04 04747, 2022.
Valentin De Bortoli, James Thornton, Jeremy Heng, and Arnaud Doucet. Diffusion schrödinger bridge with applications score- generative modeling. Advances in Neural Information Processing Systems, 34: 17695-17709, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780 8794, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Matej Grcié, Ivan Grubišić, and Simiša Segvić. Densely connected normalizing flows. Advances in Neural Information Processing Systems, 34:23968-23000 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural infor- mation processing systems, 30, 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abbeel. Flow+ Improving flow-basod generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722 2730. PMLR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47-1, 2022.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34:14745 14758, 2021.
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola. Subspace diffusion generative models. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviu, Israel, October 23-27 2022, Proceedings, Part XXIII, pp. 274-289. Springer, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXio preprint arXiv: 10196 2017.
Tero Karras, Milka Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems, 33: 12104-12114, 2020.
Dongjun Kim, Byeonghu Na, Se Jung Kwon, Dongsoo Lee, Wanmo Kang, and I1-Chul Moon. Maximum likelihood training of implicit nonlinear diffusion models. arXiv preprint arXio:2205. 13699, 2022.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances Wt neural information processing systems, 31, 2018.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in newral information processing systems, 29, 2016.
Ivan Kobyzev, Simon JD Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE tronsactions on pattern analysis and machine intelligence, :3(11):3964-399) 2020.
Zhileng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
Alex Krishevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie Yan Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. arXiv preprint arXiv:2106.06/00 2021.
Zezeng Li, Shenghao Li, Zhanpeng Wang, Na Lei, Zhongxuan Luo, and David Xianfeng Gu. Dpm-ot: A new diffusion probabilistic model based on optimal transport. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 22624 -26333, 2023.
Zuchao Li, Rui Wang, Kehai Chen, Masso Utiyama, Eicchroo Sumita, Zhuosheng Zhang, and Hai Zhao. Data-dependent gaussian prior objective for language generation. 2020.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint ar orXi::2206.0040 2022.
Lars Maalee, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2108.1609, 2021.
Eric Nalisnick and Padhraic Smyth. Stick-breaking variational autoencoders. ar preprint arXiv:1605.060 06197, 2016.
Jeeseung Park and Younggoun Kim. Styleformer: Transformer basod generative adversarial networks with style vector. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 8983 8992, 2022.
Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoen- coder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion, pp. 823 832, 2021.
Prajit Ramachandran, Tom Le Painc, Pooya Khorrami, Mohammad Babacizadch, Shiyu Chang, Yang Zhang, Mark A Hasegawa-Johnson, Roy H Campbell, and Thomas S Huang. Fast generation for convolutional autoregressive models. arXiv preprint arXiv: 1044:17040000 704.06001, 2017.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time scries forecasting. In International Conference on Machine Learning, pp. 8857-8868. PMLR, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Danilo Rezende and Shakir Mohaned. Variational inference with normalizing flows. In International con- ference on machine learning, pp. 1530- 1538. PMI.R, 2015.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the TEEE/CVE Conference on Compuler Vision and Pattern Recognition, pp. 10684 10695, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. orXiv preprint arX1v:2200.00512, 2022.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Abhishck Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. Advances in Neural Information Processing Systems, 11122333115444 2021.
Samarth Sinha and Adji Bousso Dieng. Consistency regularization for variational auto-encoders. Advances in Neural Information Processing Systems, 34:12943-12954, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. ar preprinl arXiv:2010. 02502, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438-1228 12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201.118466 2020b.
Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Variational autoencoder with implicit optimal priors. In Proceedings of the A AATT Conference on Artificial Intelligence, volume 33, pp. 5066 5073, 2019.
Yuhta Takida, Masaaki Imaizumi, Takashi Shibuya, Chieh- Hsin Lai, Toshimitsu Uesaka, Naoki Murata, and Yuki Mitsufuji. San: Inducing metrizability of gan with discriminative normalized lincar layer. arXiv preprint ar 2301 12811, 2023.
Jiayan Teng, Wendi Zheng, Ming Ding, Wenyi Hong, Jianqiao Wangni, Zhnoyi Yang, and Jie Tang. Relay dif- fusion: Unifying diffusion process across resolutions for image synthesis. arXiv preprint arXiv:2309.0833 2023.
Jakub Tomezak and Max Welling. Vae with vampprior. In International Conference on Artificial Intelli- gence and Statistics, pp. 1214-1223. PMLR, 2018.
Arash Vahdat and Jan Kautz. Nvac: A deep hierarchical variational autoencoder. Advances neural information processing systems, 33:19667 19679, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287-11302, 2021.
Vikram Voleti, Alexia Jolicoeur-Martineau, and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiw:2205.0988. 099553 2022.
Antoine Wehenkel and Gilles Louppe. Diffusion priors in variational autoencoders. orXiv preprint (tl::2106116711 2021.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXi::203..00 09481, 2022.
Mohsen Zand, Ali Etemad, and Michael Greenspan. Flow-based spatio-temporal structured prediction of motion dynamics. IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-13, 2023. doi:
Jingwei Zhang, Han Shi, Jincheng Yu, Enze Xie, and Zhenguo Li. Diffflow: A unified sde framework for score-based diffusion models and generative adversarial networks. arXiv preprint arXiv:2307.02159 2023.
Qinsheng Zhang and Yongxin Chen. Diffusion normalizing flow. Advonces in Newral Information Processing Systems, 34:16280 162911, 2021.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprinl Xiv:2220. 13902, 2022.
Qinsheng Zhang, Molei Tao, and Yongxin Chen. gddim: Generalized denoising diffusion implicit. models. arXiv preprint arXiv:2206.0055 05564,, 2022.
Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Truncated diffusion probabilistic models and diffusion- based adversarial auto-encoders. The Eleventh International Conference on Learning Representations, 2023.
17


Reviews:
[{"review_id": "kNSsfSsvNB", "reviewer": "Reviewer_gvFE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n\n- The idea of combining diffusion models with normalizing flows is new and the motivation in theory sounds plausible. \n\n- There are broad evaluations on experiments to justify the effectiveness of the idea. \n\nHowever, there are also evident cons that may be resolved to strengthen the claims and ideas of current paper. \n\nCons:\n\n- The idea of combining diffusion models with other existing methods such as GANs, VAEs, or Normalizing flows to speed up training and sampling is not new. There are many existing works in both theoretical analysis and empirical validations to demonstrates this idea. For example, in [1], the authors show empirically [1] and theoretically [2] that the sampling process could be speeded up by truncating the diffusion process and learn the truncated prior by another generative model such as GANs or VAEs. \nAlso In [2], the authors demostrates that diffusion models can be unified into other existing generative models by adjusting the SDE/ODE, which is also relevent to the current theoretic motivations of DiNof.  The authors should make comprehensive discussions on the relevance and difference to existing works. \n\n- The experiments are not persuasive enough to readers to demonstrate the superiority of current DiNof approach. In particular, the current approach shares plenty of motivations to [1], but the experimental results in table 3 Generative performance on CelebA-HQ-256 dataset and Table 4: Generative performance and sampling time on CIFAR-10, does not compare with [1]. I strongly suggest authors to add comparisons to [1] in every experiments. \n\n\n\n\nReferences\n\n[1]. Truncated Diffusion Probabilistic Models and Diffusion-based Adversarial Auto-Encoders.  https://openreview.net/forum?id=HDxgaKk956l\n\n[2]. DiffFlow: A Unified SDE for Score-Based Diffusion Models and Generative Adversarial Network. https://openreview.net/forum?id=x17qiTPDy5", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a new generative method named DiNof, which can be seen as a hybrid model between normalizing flows and diffusion models. The motivation is that the generative process of diffusion models can be speeded up by truncating the diffusion process into two stages: a linear diffusion to a noisy version of data distribution; then transforming the noisy data distribution into a prior distribution by normalizing flows. The experiments indicate the current approach can achieve a good trade-off between sample quality and sampling speed.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Section: Strengths And Weaknesses.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ULalm1yMp3", "reviewer": "Reviewer_jU2g", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. Using normalizing flow priors for diffusion models is something that has not been thoroughly investigated prior to this paper, and as such, some novelty is granted to this work. This augmentation helps reduce compute time considerably (as shown by Table 4) while attaining good performance on presented benchmarks.\n\n## Weaknesses\n\n1. Although this particular fusion of normalizing flows and diffusion models has not been investigated, the idea of using normalizing flows together with diffusion models has existed for some time; specifically since DiffFlow was presented by Zhang & Chen in 2021. To the paper's credit, they are upfront about this and introduce Figure 2 explicitly to distinguish their fusion from previously existing work. That being said, this does take away from some of the novelty, especially given that the fusion is not of a very complex nature (i.e. the paper is focused on presenting a diffusion model with a less trivial prior to reduce computational load).\n\n2. The results for this paper are good, but the paper makes the misleading claim that several of the results are state-of-the-art. In particular, they claim that their approach \"obtains a state-of-the-art FID of 7.11\" on the CelebA-HQ-256 dataset. Although this is a solid result, there are at least 6 baselines with a better FID score than this (please see reference [a]). Some of these results date back to 2021, i.e., they are not that recent. Additionally, there are at least 20 baselines with a better FID than 2.01 on CIFAR-10 (please see reference [b]). The best of these baselines should be included in this paper for both results to avoid giving the reader a misleading representation about the strength of the method. All claims made about the state-of-the-art nature of the presented results should be weakened considerably.\n\n## Verdict\n\nThis paper presents and investigates the fusion of using diffusion models with normalizing flow priors. This particular fusion of diffusion models and normalizing flows is novel, although the idea of combining diffusion models and normalizing flows has been studied before (Zhang & Chen, 2021). The paper's presented method performs well on the CIFAR-10 and CelebA-HQ-256 benchmarks, although the paper makes misleading claims about the state-of-the-art nature of their results. In particular, there are at least 6 papers with a better CelebA-HQ-256 result (see reference [a] below). That being said, the authors' approach does considerably reduce computational load while obtaining solid results. As such, I think this work would be interesting to some portion of TMLR's audience, however the claims made in the submission about the state-of-the-art nature of the results are not well supported and this language needs to be changed multiple times throughout the paper. Given that both criteria for acceptance to TMLR are not satisfied (in particular the papers' claims about their state-of-the-art results), I must recommend a reject rating for the paper in its current state.\n\n### References\n\n[a] Papers with Code: Image Generation on CelebA-HQ 256x256. https://paperswithcode.com/sota/image-generation-on-celeba-hq-256x256 \n\n[b] Papers with Code: Image Generation on CIFAR-10. https://paperswithcode.com/sota/image-generation-on-cifar-10", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new generative model, based on the joint fusion of a diffusion model and normalizing flow. The authors refer to their method as DiNof (which stands for Diffusion with Normalizing Flow Priors). In essence, the core proposition is the use of a data-dependent, deterministic normalizing flow prior as an alternative to the random noise currently used in the context of standard diffusion models. They claim this enables more accurate modelling of complex distributions and with a smaller number of sampling steps. These claims are validated on two datasets: CIFAR-10 and CelebA-HQ-256. DiNof attains FID scores of 2.01 and 7.11 on the CIFAR-10 and CelebA datasets, respectively.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned above in the weaknesses section, I believe the baselines in this paper are incomplete. In particular, the strongest baselines for CIFAR-10 and CelebA-HQ-256 from references [a] and [b] should be added to the paper, and all claims made regarding the state-of-the-art FID results should be rewritten/weakened.\n\nIn addition, a non-exhaustive list of minor corrections is given below:\n\n- Section 1, third paragraph: \"single-step at inferences\" -> \"single-step at inference\"\n- Section 2, first paragraph: \"an VAE\" -> \"a VAE\"\n- Section 2, fifth paragraph: \"a significant disadvantage of diffusion modeles\" -> \"a significant disadvantage of diffusion models\"\n- Section 3, second paragraph: \"remarks on the diffusion models and normalizing flows\" -> \"remarks on diffusion models and normalizing flows\"\n- Section 3, final paragraph: \"It can interpolate between deterministic and the stochastic processes.\" -> \"It can interpolate between deterministic and stochastic processes.\"\n- Section 4.2.2, first paragraph: Please use \\citet for in-text citations (e.g. for De Bortoli et al., 2021).\n- Section 4.2.2, third paragraph: \"it achieves 0.19 increase in the FID\" -> \"it achieves a 0.19 decrease in the FID\" or \"it achieves a 0.19 improvement in the FID\"\n- Section 4.2.2, fourth paragraph: \"To save the computations,\" -> \"To save on computation,\"\n- Section 4.2.3, first paragraph: \"Thus, we employ a considerably less number of diffusion/sampling steps\" -> \"Thus, we employ a considerably smaller number of diffusion/sampling steps\"", "broader_impact_concerns": "This work deals with increasing the efficacy of generative models. The potential ethical ramifications are similar to those of any modern generative AI method. A brief impact statement should be added to clarify this and highlight the potential adverse effects of generative AI.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WhauCWHxMu", "reviewer": "Reviewer_GwNV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The idea is simple and intuitive. \n- The experimental results are good, indicating good FID and IS scores, while being faster than DDIM and DDPM.\n\n## Weaknesses\n- Actually, the idea is not novel, because there are some similar works [1] proposed to learn generators to map from $x(T)$ to $x(T_m)$. However, I do not make a decision mainly based on the novelty because novelty is not a main criterion to justify a TMLR paper.\n- It would be great if the authors can discuss what the benefit of learning a normalizing flow to map from $x(T)$ to $x(T_m)$ rather a generator.\n\n[1] Li Z, Li S, Wang Z, Lei N, Luo Z, Gu DX. DPM-OT: A New Diffusion Probabilistic Model Based on Optimal Transport. InProceedings of the IEEE/CVF International Conference on Computer Vision 2023 (pp. 22624-22633).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to use a normalizing flow to speed up the inference process of a diffusion model. Specifically, it learns a normalizing flow to map from the distribution of $x(T)$ to that of $x(T_m)$ where $T_m <T$. In inference, it uses the normalizing flow to infer from $x(T)$ (i.e., a Gaussian noise) to $x(T_m)$ and then uses the diffusion model to further infer to an image.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- More discussions to distinguish your work and DiffFlow (Zhang & Chen, 2021).\n- Please discuss and compare with [1] in experiments.", "broader_impact_concerns": "There are no concerns on the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **General**:
* The paper proposes a new generative model, namely, a combination of a diffusion model and a normalizing flow.  The main idea is to use a data-dependent, deterministic normalizing flow prior as an alternative to the random noise typically used in the context of vanilla diffusion models.

**Pros**:
+ The idea of combining diffusion models with normalizing flows is new and the motivation in theory sounds plausible.
+ There are broad evaluations of experiments to justify the effectiveness of the idea.

**Cons**:
- The idea of combining diffusion models with other existing methods such as GANs, VAEs, or Normalizing flows to speed up training and sampling is not necessarily new. Therefore, a better discussion is needed.
- The experiments are not persuasive enough and miss some relevant baselines. Therefore, new experiments and comparisons are needed.

**Rebuttal**:
* The authors were active in the rebuttal phase.
* They authors updated the paper according to all requests provided by the reviewers. Therefore, the paper could be accepted without further revision rounds.
Decision: Accept (poster)