DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents
Authors: Kushagra Pandey
Avideep Mukherjee
Piyush Rai
Abhishek Kumar
Venue: Accepted by TMLR ()

Abstract: Diffusion probabilistic models have been shown to generate state-of-the-art results on several competitive image synthesis benchmarks but lack a low-dimensional, interpretable latent space, and are slow at generation. On the other hand, standard Variational Autoencoders (VAEs) typically have access to a low-dimensional latent space but exhibit poor sample quality. We present DiffuseVAE, a novel generative framework that integrates VAE within a diffusion model framework, and leverage this to design novel conditional parameterizations for diffusion models. We show that the resulting model equips diffusion models with a low-dimensional VAE inferred latent code which can be used for downstream tasks like controllable synthesis. The proposed method also improves upon the speed vs quality tradeoff exhibited in standard unconditional DDPM/DDIM models (for instance, \textbf{FID of 16.47 vs 34.36} using a standard DDIM on the CelebA-HQ-128 benchmark using \textbf{T=10} reverse process steps) without having explicitly trained for such an objective. Furthermore, the proposed model exhibits synthesis quality comparable to state-of-the-art models on standard image synthesis benchmarks like CIFAR-10 and CelebA-64 while outperforming most existing VAE-based methods. Lastly, we show that the proposed method exhibits inherent generalization to different types of noise in the conditioning signal. For reproducibility, our source code is publicly available at \url{https://github.com/kpandey008/DiffuseVAE}.

Published in Transactions on Machine Learning Research (11/2022)
DiffuseVAE: Efficient, Controllable and High-Fidelity Gener- ation from Low-Dimensional Latents
Kushagra Pandey
pandeyk1@nce @ruci
Deportment of Computer Science University of California, Irvine
Avidcep Mukherjee
avideep@cse. wwtp::wwwss.....ccoooo www.kikkkkkkk........o....o
Department of Computer Science Indian Institute of Technology, Kanpur
Piyush Rai
pigush@cse.iikkcom
Department of Computer Science Indian Institute of Technology, Kanpur
Abhishek Kumar
abhishk@google.com
Google Research, Brain Team
Reviewed OpenReview: https: https:://ppenreviau
openreview. net/ forum? id= LdggooNRRLL
Abstract
Diffusion probabilistic models have been shown to generate statc-of-the-art results on sev- eral compelitive image synthesis benchmarks but lack a low-dimensional, interpretable latent space, and are slow at generation. On the other hand, standard Variational Autoencoders (VAEs) typically have access to a low-dimensional latent space but exhibit poor sample quality. We present DiffuseVAE, a novel generative framework that integrates VAE within diffusion model framework, and leverage this to design novel conditional parameteriza- tions for diffusion models. We show that the resulting model equips diffusion models with a low-dimensional VAE inferred latent code which can be used for downstream tasks like controllable synthesis. The proposed method also improves upon the speed vs quality trade- off exhibited in standard unconditional DDPM DDPM/DDIM /DDIM modols (for instance, FID 16.47 vs 34.36 using a standard DDIM on the CelebA-HQ-122 benchmark using T=10 reverse process steps) without having explicitly trained for such an objective. Furthermore, the proposed model exhibits synthesis quality comparable state-of-the-art models on stan- dard image synthesis benchmarks like CIFAR-10 and CelebA-64 while outperforming most existing VAF-based methods. Lastly, we show that the proposed method exhibits inherent generalization to different types of noise in the conditioning signal. For reproducibility, our source code is publicly available at htttps///iitee cwm comm//paaddyyy0///iffuss////
1 Introduction
Generative modeling is the task of capturing the underlying data distribution and learning to generate novel samples from posited explicit/impp /implicit distribution of the data in unsupervised manner. Variational Autocncoders (VAEs) (ingma & Welling 2014 Rezende & Mohamed] 2016) are a type of explicit-likelhoodd based generative models which are often also used to learn a low-dimensionall latent represenlation for the data. The resulting framework very flexible and can be used for downstream applications, such as learning disentangled represontations (Higgins et al. 2017 Chen al.] 2019 Burgess et al.) 20188) semi-supervised learning (Kingma al. 20144) anomaly detection (Pol al. (020), among others. However, in image synthesis applications, VAE generated samples (or reconstructions) usually blurry and fail to incorporate

---

Published in Transactions on Machine Learning Research (11/2022)
EI
Figure 1: DiffuseVAF, generated samples on the CelebA-HQ-256 (Left), CelebA-HQ-128 (Middle), CTFAR- 10 (Right, Top) and CelebA-64 (Right, Bottom) datasets using just 25, 10, 25 and 25 time-steps in the reverse process for the respective datasets. The generation is entirely driven by low dimensional latents the diffusion process latents are fixed and shared between samples after the model is trained (See Section for more details).
high-frequency information (Dosovitskiy & Brox 2016 Despite recent advances (van den Oord et al. 2018; Razavi et al. 2019
Xiao al. 2021 in improving VAE sample quality, most VAE-based methods require large latent code hierarchies. Even then, there is still a significant gap in
Vahdat & Kautz, 2021 Child 2021
sample quality between VAEs and their implicit-likeliben counterparts like GANs [Goodfellow et al. 2014
Karras et al. 2018
2019
2020b).
In contrast, Diffusion Probabilistic Models (DDPM) (Sohl-Dickster et al. 2015 Ho et al. 2020) have been
shown to achieve impressive performance on several image synthesis benchmarks, even surpassing GANs on several such benchmarks (Dhariwal & Nichol 2021 Ho et al. 2021 However, conventional diffusion models require expensive iterative sampling procedure and lack a low-dimensional latent representation, limiting these models' practical applicability for downstream applications.
We present DiA, a novel framework which combines the best of both VAEs and DDPMs in an attempt to alleviate the aforementioned issues with both types model families. We present a novel two- stage conditioning framework where, in the first. stage, any arbitrary conditioning signal (y) can be first modeled using standard VAE. In the second stage, we can then model the training data (x) using a DDPM conditioned y and the low-dimensional VAE latent code representation of y. With some simplifying design choices, our framework reduces to generulor-refiner framework which involves fitting VAE on the training data (x) itself in the first stage followed by modeling x in the second stage using DDPM conditioned on the VAE reconstructions (æ) of the training data,, The main contributions of our work can be summarized as follows:
A novel conditioning framework: We propose a generic DiffuseVAE conditioning framework and show that our framework can be reduced to a simple generator-refiner framework which blurry samples generated from a VAE are refined using conditional DDPM formulation (See Fig 2 This effectively equips the diffusion process with a low dimensional latent space. As a part of conditioning framework, explore two types of conditioning formulations in the second stage DDPM model.
2. Controllable synthesis from low-dimensional latent: We show that, as part of our model design, major structure in the DiffuseVAE, generated samples can be controlled directly using the low-dimensional VAE latent space while the diffusson process noise controls minor stochastic details in the final generated samples.
Better speed vs quality tradeoff: We show that DiffuseVAA inherently provides a better speed vs quality tradcoff as compared to standard DDPM model on several image benchmarks. Morcover, combined with DDIM sampling (Song et al.] (0211)). the proposed model can generate plausible samples in less as 10 reverse process sampling steps (For example, the proposed method achieves
2

---

Published in Transactions on Machine Learning Research (11/2022)
an FID (Heusel et al. 2018 of 16.47 as compared to 34.36 by the corresponding DDIM model T=10 steps on the HQQ benchmark (Karras et al. 2018)).
State of the art comparisons: We show that DiffuseVAE exhibits synthesis quality comparable recent state-of-the-art on standard image synthesis benchmarks like CIFAR-10 (Krizhevsky 2009).
CelebA-64 (Liu et al. 2015 and CelebA-HQ (Karras et all 2018) while maintaining access to
low-dimensional latent code representation.
Generalization to different noises in the conditioning signal: We show that pre-trained DiffuseVAE model exhibits generalization to different noise types in the DDPM conditioning signal exhibiting the effectiveness of our conditioning framework.
2 Background
2.1 Variational Autoencoders
VAEs (Kingma & Welling 2014 Rezende & Mohamed 2016) based on simple but principled encoder- decoder based formulation. Given data z with a latent representation 2, learning the VAE is done by maximizing the evidence lower bound (ELBO) on the data log-iikel,oood, logp(z) (which is intractable to compute in general). The VAE optimization objective can be stated as follows
(1) Under amortized variational inference, the approximate posterior on the latents, i.e., (10(2|x)), and the likelihood (pa(z|²)) distribution can be modeled using deep neural networks with parameters o and 0, respectively, using the reparaneteriiztion trick (Kingma & Welling 2014 Rezende Mohamed 20166). The choice of the prior distribution p(2) is flexible and can vary from standard Gaussian (Kingma & Welling. 2014) to more expressive priors (van den Berg et al. 2019
[(0, 6) = [1o(==))[oggg) (o - Dx (xr.lo(=//>)ll
Grathwohl al. 2018 Kingma al. 2017).
2.2 Denoising Diffusion Probabilistic Models
DDPMs (Sohl-Dickstein al. 2015 Ho et al. 2020) are vatent-variable models consisting of a forward noising process (q(x)))(=)) (2o)) which gradually destroys the structure of the data xo and reverse denoising process ((((zor)) which learns to recover the original data zo from the noisy input. The forward noising process is modeled using a first-order Markov chain with Gaussian transitions and is fixed throughout training, and the noise schedules 31 to 3T can be fixed or learned. The form of the forward process can be summarized as follows:
T (fir)))) = II ((21(111))
(2)
1=1
(( N( B1x1-1,
(3)
(<<e/oo) = (VVa (1 - a.) where at = - (3) and ā. II
(4)
The reverse process also be parameterized using first-order Markov chain with learned Gaussian transition distribution as follows
p(Zo.T) = p(xr) II
po (
(5)
po = ((o((a, 1), Eo(xr,)) ())
(6)
Given large enough and well-behaved variance schodule of Br, the distribution g( (*)) will approximato an isotropic Gaussian. The entire probabilistic system can be trained end-to-eni using variational inference. During sampling, sample can be generated from the underlying data distribution by sampling latent
3

---

Published in Transactions on Machine Learning Research (11/2022)
p
22
2o
Encoder
Decoder
#7-1
PT-2
øT-5
JO
Stage-1 VAE Training
Stage-2 DDPM Training
Figure 2: Proposed DiffuseVAE generative process under the simplifying design choices discussed in Section DiffuseVAE is trained in two-stage manner: The VAE encoder takes the original image as input and generates a reconstruction to which is used to condition the second stage DDPM.
3.2
(of the same size the training data point mn) from p(ær) (chosen to be isotropic Gaussian distribution) and running the reverse process. We highly encourage the readers to refer to Appendix A for more detailed background on diffusion models.
3
DiffuseVAE: VAEs meet Diffusion Models
3.1 DiffuseVAE Training Objective
Given a high-resolution image wo, auxiliary conditioning signal y to be modelled using VAE, latent representation associated with y, and sequence of T representations Fi.T learned by diffusion model, the DiffuseVAE joint distribution can be factorized as:
p(Zo:r: y, = p(=))p((yl)))))) (pppyll
where 0 and are the parameters of the VAE decoder and the reverse process of the conditional diffusion model, respectively. Furthermore, since the joint posterior p(z1.T, 2ly, 20) is intractable to compute, we approximate it using surrogate posterior =ly, which can also be factorized into the following conditional distributions:
((E1r zly, mo) = qo (2ly, xo (1)) 2,
(8)
where vb the parameters of the VAE recognition network (gos (o)) As considered in previous works (Sohl-Dickstein et al.] 2015; Ho et al. 2020 we keep the DDPM forward process (9(#1:1 ly. 2, xo)) non- trainable throughout training. The log- likellhoood of the training data then be obtained as:
logp(#o, v) log / p( v, 2)
Since this estimate intractable estimate analytically, we optimize the ELBO corresponding the log- likelihood. It can be shown that the log-likelihood estimate of the data be approximated using the following lower bound (See Appendix D.1 for the proof)
logp(zo:1) v) 2 E (o [p) -
+
LVaR
Dol (1,
(10)
E Bang(=)y,
E (1.,,,))
g(x 20)
CDoPM
We next discuss the choice of the conditioning signal y, some simplifying design choices and several param- eterization choices for the VAE and the DDPM models.
4

---

Published in Transactions on Machine Learning Research (11/2022)
3.2 Simplifying design choices
In this work interested in unconditional modeling of data. To this end, make the following simplifying design choices:
1. Choice of the conditioning signal yf We assume the conditioning signal to be itself which ensures a deterministic mapping between y and To- Given this choice, we do not condition the reverse diffusion process on y and take it as pa (For)>z) in Eq. 10
Choice of the conditioning signal 2: Secondly, instcad conditioning the reverse diffusion directly on the VAE inferred latent code 2, we condition the second stage DDPM model on the VAE reconstruction fo which a deterministic function of z.
Two-stage training: We train Eq. 10 sequential two-stage manner, i.e., first optimizing LVAE and then optimizing for LDDPM in the second stage while fixing and th (i.e. freezing the VAE encoder and the decoder).
With these design choices, shown in Fig. 2 the DiffuseVAE, training objective reduces to simply training VAE model on the training data 2o in the first stage and conditioning the DDPM model the VAE reconstructions in the second stage. We next discuss the specific parameterization choices for the VAE and DDPM models. We also justify these design choices in Appendix
3.3
VAE parameterization
In this work, we only consider the standard VAE (with single stochastic layer) as discussed in Section 2.1 However, in principle, due to the flexibility of the DiffuseVAE two-stage training, more sophisticated, multi-stage VAE approaches also be utilized to model the imput data Zo. One caveat of using multi-stage VAE approaches is that we might no longer have access to the useful low-dimensional representation of the data.
proposed in (Razavi et al.) 2019 Child 2021 Vahdat & Kautz 2021)
3.4
DDPM parameterization
In this section, we discuss the two types of conditional DDPM formulations considered this work.
3.4.1 Formulation 1
In this formulation, we make the following simplifying assumptions
1. The forward process transitions are conditionally independent of the VAE reconstructions å and the latent code information z í.c. ((11r12, xo) x q(x1:)/20).
The reverse process transitions are conditionally dependent on only the VAE reconstruction, i.c., p(zor1|2) ss p(zorr|00)
A similar parameterization has been considered in recent work conditional DDPM models (Ho et al. 2021 Saharia et al. 2021). We concatenate the VAE reconstruction to the reverse process representation at each time step f to obtain 21-1-
3.4.2 Formulation 2
In this formulation, we make the following simplifying assumptions
The forward process transitions are conditionally dependent on the VAE reconstruction, i.c., (1111122,00) q(aie 20)

---

Published in Transactions on Machine Learning Research (11/2022)
(a) Formulation-1
(b) Formulation-2
Figure 3: ralustration of the generalor-refiner framework in DiffuseVAE. The VAE generated samples (Boltom row) are refined by the Stage-2 DDPM model with T=1000 during inference (Top Row).
2. The reverse process transitions conditionally dependent only the VAE reconstruction, i.c., p(z))r|z) ss p(zo.r(ã0)
Specifically, we design the forward process transitions to incorporate the VAE reconstruction åg as follows:
((21/20, 2o) = -N(V -1000 + 20.311)
(11)
(10))))11,00)) = NNW--NWW/1/-B ( B, + (1- (1)1-1))0.. B.1) for 1>
It can be shown that the forward conditional marginal in this case becomes (See Appendix D.2 for proof)
(<re/=a, 2b) = -N(Va + 20. T.)
(12)
For t = T and well-behaved noise schedule Bt, âr 0 which implics ((0r|20 fo) = Wi I). Intuitively, this means that the Gaussian N(ūu, becomes our base measure (p(zr)) during inference which we need to run our reverse process. Since the simplified denoising training formulation proposed in (Ho et al. 2020) depends on the functional form of the forward process posterior Q(=;-1/T,, 20), this formulation results in several modifications the standard DDPM training and inference which are discussed in Appendix
4 Experiments
We now investigate several properties of the DiffuseVAE model. We use a mix of qualitative and quan- titative ovaluations for demonstrating these propertics on several image synthesis benchmarks including CIFAR-10 (Krizhevsky 2009), CelebA-64 (Liu al. 2015), CelebA-HQ (Karras et al., 2018) and LHQ-256 Skorokhodov al. 2021) datasets. For quantitative evaluations involving sample quality, we use the FID Heusel al. 2018)) metric. We also report the Inception Score (IS) metric (Salimans et al. 2016) for state- CIFAR-10. For all the experiments, (7) to 1000 during training. The noise schedule in the DDPM forward process was set to linear schedule between 3 = 10- and B2 = 0.02 during training, More details regarding the model and training hyperpa- rameters can be found in Appendix Some additional experimental results are presented in Appendix
of-the-art comparisons
set the number of diffusion time-steps
4.1 Generator-refinet framework
Fig. 3 shows samples generated from the proposed DiffuseVAE model trained on the CelebA-HQ dataset at the 128 x 128 resolution and their corresponding Stago-1 VAE samples. For both DiffuseVAE formulations-1 and 2, DiffuseVAE generated samples (Fig. 3 (Top row) are a refinement of the blurry samples generated by single-stage VAE model (Bottom row).

---

Published in Transactions on Machine Learning Research (11/2022)
Non interplaaions
interpolations
Figure 4: DiffuseVAE samples generated by linearly interpolating in the VAE latent space (Formulation- 1, T=0000). denotes the interpolation factor. Middle row: VAE generated interpolation between two samples. Top row: Corresponding DDPM refinements for VAE samples in the Middle Row. Bottom row: DDPM refinements for VAE samples in the Middle Row with shared DDPM stochasticity among all samples.
FID@10k |
Baseline VAE
87.28
Baseline VAE + DDPM Refiner (Form-1)
10.87
Baseline VAE + DDPM Refiner (Form-2)
11.44
Table 1: Quantitative Illustationn of the gencrator-refiner framework in DiffuseVAE for the CelebA-HQ (128 x 128) dataset. FID reported on 10k samples (Lower better)
This observation qualitatively validates our generutor-refnnr framework in which the Stage-1 VAE model acts as generator and the Stage-2 DDPM model acts as refiner. The results in Table D quantitatively justify this argument where the CelebA-HQ--228 benchmark, DiffuseVAE improves the FID score of baseline VAE by about eight times. Additional qualitative results demonstrating this observation can be found in Fig. 13
4.2
Controllable synthesis via low- low-dimensional DiffuseVAE latents
4.2.1
DiffuseVAE Interpolation
The proposed DiffuseVAE model consists of two types of latent representations: the low-dimensional VAE latent code 2oae and the DDPM intermediate representations I1.r associated with the DDPM reverse process (which are the same size of the input image zi and thus might not be beneficial for downstream tasks). We next discuss the effects of manipulating both zeee and æy. Although, it possible to inspect interpolations the intermediate DDPM representations T1:T-1, do not investigate this case in this work. We consider the following interpolation settings:
Interpolation in the VAE latent space Zvee: We first sample two VAE latent codes (1) and (2) using the standard Gaussian distribution. We then perform lincar interpolation between 2nae and zode to obtain intermediate VAE latent codes E = A=vae + (1 X)eee for (0 X < 1), which are then used to generate the corresponding DiffuseVAE samples.
Fig. g (Middle Row) shows the VAE samples generated by interpolating between two sampled VAE codes as described previously. The corresponding DiffuscVAE, generated samples obtained by interpolating in the Zvae space are shown in Fig. | (Top Row). It can be observed that the refined samples corresponding to the blurry VAE samples preserve the overall structure of the image (facial expressions, hair style, gender etc).

---

Published in Transactions on Machine Learning Research (11/2022)
VAE Samples
A 1.0
Figure DiffuseVAE samples generated by lincarly interpolating in the ar latent space (Formulation-1, T=1000). denotes the interpolation factor.
However, due to the stochasticity in the reverse process sampling in the second stage DDPM model, minor image details (like lip color and minor changes in skin tone) do not vary smoothly between the interpolation samples due to which the overall interpolation is not smooth. This becomes more clear when interpolating the DDPM latent Pr while keeping the VAE code eyae fixed as discussed next.
Interpolation in the DDPM latent space with fixed Zvac: Next, we sample the VAE latent code 2eee using the standard Gaussian distribution. With a fixed Zuee, we then sample two initial DDPM represen- tations zf and IT from the reverse process base mcasure p(rr). We then perform linear interpolation (1) between 2y and o, with fixed Zeae to generate the final DiffuseVAE samples (Note that interpola- tion is not performed on other DDPM latents, 21.7, which are obtained using ancestral sampling from the corresponding ir's 's as usual).
Fig 3 shows the DiffuseVAE generated samples with a fixed Zoee and the interpolated æT. As can be observed, interpolating in the DDPM latent space leads to changes in minor features (skin tone, lip color, collar color etc.) of the generated samples while major image structure (face orientation, gender, facial expressions) is preserved across samples. This obscrvation implics that the low-dimensional VAE latent code mostly controls the structure and diversity of the generated samples and has more entropy than the DDPM representations æT, which carry minor stochastic information. Moreover, this results in non-smooth DiffuseVAE interpolations. We discuss a potential remedy next.
Handling the DDPM stochasticity: The icct in the second stage DDPM sampling process can occasionally result in artifacts in DiffuseVAE samples which might be undesirable downstream applications. To make the samples generated from DiffusoVAE deterministic (i.c. controllable only from 2yee), we simply share all stochasticity in the DDPM reverse process (i.e. due to ØT and 24) across all generated samples. This simple technique adds more consistency in latent interpolations Row) while also enabling deterministic sampling. This observation is intuitive as initializing the second stage DDPM in DiffuseVAE with different stochastic noise codes during sampling might be understood as imparting different styles to the refined sample. Thus, sharing this stochasticity in DDPM sampling across samples implics using the same stylization for all refined samples leading to smoothness between interpolations. Having achieved more cousistency in our interpolations, we can now utilize the low-dimensional VAE latent code for controllable synthesis which we discuss next.
be observed in Fig. 4 (Bottom
4.2.2 From Interpolation to Controllable Generation
Since DiffuseVAE gives access to the entire low dimensional VAE latent space, we perform image manipulation by performing vector arithmetic in the VAE latent space (See Appendix G.2 for details). The resulting latent code can then be used to sample from DiffuseVAE to obtain refined manipulated image. As discussed in the previous section, we share the DDPM latents across samples to prevent the generated samples from using different styles. Fig. demonstrates single-attribute image manipulation using DiffuseVAE on several attributes like Gender, Age and Hair texture. Morcover, the vector arithmetic in the latent space can be composed to generate composite edits (See Fig. 6). thus signifying the usefulness of low-dimensional latent code representation. Some additional results image manipulation illustrated in Fig. 14
8

---

Published in Transactions on Machine Learning Research (11/2022)
Original
Male
Young
Black Hair
Receding
Original
Black Hair
Male
Young
Eyeglasses
Hairline
Figure Controllable generation on DiffuseVAE generated samples on the CelebA-HO, 256 dataset. Red and green arrows indicate vector subtract and addition operations respectively. Top and Bottom panels show single edits and composite edits respectively.
4.3 Better Sampling Speed vs Quality tradeoffs with DiffuseVAE
There exists a trade-off between the number reverse process sampling steps vs the quality of the generated samples in DDPMs. Usually the best sample quality is achieved when the number reverse process steps used during inference matches the number of time-steps used during training. However, this can be very time-consuming (Song al.) 2021a). On the other hand, the number of reverse process steps is reduced, the sample quality gets worse. We next examine this trade-off in detail.
Comparison with a baseline unconditional DDPM: Table 2 compares the sample quality (in terms of FID) vs the number of sampling steps between DiffuseVAE and our unconditional DDPM bascline on the CelebA-HQQ-12 dataset. For all time-steps T 10 to = 100, DiffuseVAE outperforms the standard DDPM by large margins terms of FID. Between DiffuseVAF formulations, the sample quality similar with Formulation-1 performing slightly better. More notably, the FID score of DiffuseVAE T 25 and 50 is better than that of unconditional DDPM at T = 50 and 100 respectively. Thus, in low time-step regimes, the speed vs quality tradeoff in DiffuseVAF. is significantly better than an unconditional DDPM baseline. It is worth noting that this property is intrinsic to DiffuseVAE as the model was not specifically trained to reduce the number of reverse process sampling steps during inference (Salimans & Ho 2022
However, at T 1000 the unconditional DDPM baseline performs better than both DiffuseVAR formulations- and We hypothesize that this gap in performance can be primarily attributed the prior-hole problem, i.e., the mismatch between the VAE prior p(=) and the aggregated posterior q(z) (Bauer & Mnih 2019 Dai & Wipf 2019 Ghosh et al. 2020) due to which VAEs generate poor samples from regions of the latent space unseen during training. DDPM refinement. of such samples can affect the FID scores negatively. We confirm this hypothesis next.
Improving DiffuseVAE sample quality using post-fitting: One way to alleviate the prior-hole problem is fit a density estimator (denoted by Ex-PDE) on the training latent codes and sample from this estimator during inference in (van den Oord al. 2017 Razavi et al. 2019 Ghosh et al., (020). Along similar lines, fit. GMM the VAE latent code representations of the training data. We then use this estimator to
9

---

Published in Transactions on Machine Learning Research (11/2022)
10
25
50
100
1000
DDPM (Uneond)
41.25
27.83
21.40
16.29
8.93
DiffuseVA (Form-1)
19.44
13.68
12.63
DifluseVAE (Form-2)
31,08
19.67
13.96
13,96
13.20
DiffuseVAE (Form-1, GMM=100)
30.74
18.55
14.10
12.12
10.87
DifluseVAE (Form-2, GMM=100)
30.66
18.98
14:45
12.50
11.44
Table 2: Comparison of sample quality (FID@10K) vs speed on the CelebA-HQ-112 dataset (DiffuseVAB
unconditional DDPM). Top Row represents the number of reverse process sampling steps.
CelebAHQ- 1228
CelebA-61
10
25
50
100
10
25
50
100
DDIM (uncond)
34.36
25.04
19.83
16.69
14.14
7.88
6.77
6.38
DiffuseVAE (Form-1)
19.42
14.53
10.79
6.87
6.08
5.82
DiffuseVAE (Form-1, Ex-PDE)
18.01
13.21
12.40
12.28
10.44
6.59
5.81
5.55
DiffusVAE (Form-2)
17.51
13.45
12:36
12.51
9.81
6.34
5.83
5.59
DiffuseVAE (Form-2, Ex-PDE)
16.47
11.62
10.83
10.28
9.56
5.90
5.43
5.21
Table 3: Comparison of sample quality (FID&10k) vs speed between DiffuseVAE and the unconditional DDIM on the CelebA-HQ-123 and CelebA-64 datasets. DiffuseVAE with Form-2 shows a better speed- quality tradeoff than Form-l. Overall, DiffuseVAE achieves upto 4x and 10x speedups on the CelebA-64 and the CelebA-HQ-12 HQ8128 datasets respectively as compared to the uncondtional DDIM
sample VAE latent codes during DiffuseVAE sampling. Table the FID scores on the CelebA-HQ-L dataset for both DiffuseVAE formulations using a GMM with 100 components. Across all time-steps, using Ex-PDE during sampling leads to reduced gap in sample quality at T 1000, thereby confirming our hypothesis. We believe that the remaining gap be closed by using stronger density estimators which we do not explore in this work. Morcover, side benefit of using a Ex-PDE during sampling is further improvement in the speed-quality tradeoff.
Further improvements with DDIM: DDIM (Song et al. 2021a employs a non- Markovian forward process and achieves a better speed-quality tradeoff than DDPM along with deterministic sampling. Since DiffuseVAE employs DDPM model in the refiner stage, we found DDIM sampling to be complementary with the DiffuseVAE framework. Notably, since the forward process for DiffuseVAE (Form-2) is different, we derive the DDIM updates for this formulation in Appendix B Table 3 compares the speed-quality tradeoff between DDIM and DiffuseVAE (with DDIM sampling) on the CelebA-HQ- 128 and CelebA-64 datasets. DiffuseVAE (both formulations) largely outperforms the standard unconditional DDIM at all time-steps. For the CelebA-HQ-128 benchmark, similar to our previous observation, DiffuseVAE (with DDIM sampling and Ex-PDE using GMMs) al T 25 and steps performs better than the standard DDIM T 50 and 100 steps respectively. In fact, T 10, DiffuseVAE (with Formulation-2) achieves a FID of 16.47 which is better than DDIM with T 100 steps, thus providing speedup of almost 10x. Similarly for the CelebA- 64 benchmark, at T 25, DiffuseVAE (Formulation-2) performs similarly to the unconditional DDIM T = 100, thus providing 4x speedup. Lastly, it can be observed from Tables 3 11 and 12 that in the low time-step regime, DiffuseVAA (Form-2) usually performs better than Form-1 and that the speed-quality trade-off in DiffuseVAE becomes better with increasing image resolutions.
4.4 State-of-the-ar comparisons
For reporting comparisons with the state-of-the-art we primarily use the FID HHuueel et al. 2018 metric to assess sample quality. We compule FID on 50k samples for CIFAR-10 and CelebA-64. For comparisons on the CelebA-HQ-255 dataset, we report the FID only for 10k samples (as opposed to 30k samples which is the norm on this benchmark) due to compute limitations. Due to this, we anticipate the true FID score on this benchmark using our method be lower. However, as we show, the FID score obtained by DiffuseVAE this benchmark on 10k samples is still comparable to state-of-the-art.
10

---

Published in Transactions on Machine Learning Research (11/2022)
Method
FID&50k
IS
DiffuseVA (Form-1, 1000000)
2.95
9.60
0.11
DiffuseVAE (Formn-2 T:10000 T=10000
2.86
9.59 1: 0.13
DiffuseVA (Form-1; 1, T= 1000, GMM=50)
9.51
Ours
DiffuseVAE (Form-2, T=1000, GMM=50)
2.80
9.51 0.08
DillueVAE--225 (Form-2, T=1000, GMM=50)
2.62
9.75 0.08
DDPM (T=1000, Our impl.)
3.01
9.35
0.16
VAE Bascline
139.50
3.23 1: 0.02
VAE Baseline (GMM=50)
137.08
3.30
0.02
VAEBM (Nian
2021
(w/ PC)
12.19
8.43
VAE-based
DC- VAT [ Faarmar
2021
17.90
8.2
methods
NVAE (Vandat & Kantz
2021
51.67
5.51
NCP-VXE AAneja
24.08
LSGM (FID) Valrial et 2021
2.10
D2C (Sinba et al
10.15
AutoGAN (Cau et al.
2020
12.4
8.55 0.1
ProGAN (Karras
2015)
15.52
8.56 0.10
GAN-based methods
StyleGAN2 (w/o ADA) TKarra et al
2019
8.32
9.21 221100 0.09
StyleGAN2-ADA (Karra 2018
20208
9.83
SNGAN (Miyato of al SNGAN + DDLS Che et
21.7
8.22 0.05
15.42
9.09 : 0.10
NCSN (Song & Ermon 2020a NCSNv2 (w/ /eddoiiingg Song
25.32
8.87 + 0.12
2020b
8.40
Score-based methods
DDPM (Ho o5 al. 2020
3.17
9.46 + 0.1
SDE (NCSN- (song el 61
2021b
2.45
9.73
SDE (DDPM++)
2021D
2.78
9.64
Table 4: Generative performance unconditional CIFAR-10. FID and TS computed 50k samples
Table 4 shows quantitative comparison between DiffuseVAR. and other state-of-the-art unconditional gen- erative models in terms of sample quality (F1D@50k) and sample diversity (IS) on the CIFAR-10 dataset. Interestingly, our unconditional DDPM bascline achieves better FID scores on CIFAR-10 than reported in (Ho et al. 2020p. DiffuseVAR. clearly outperforms the DDPM baseline (with and without Ex-PDF) in terms of FID while maintaining a competitive IS score with continuous score based methods indicating good sam- ple diversity. Notably, with the exception of LSGM (Vahdat et al. 2021), DiffuseVAB outperforms all prior state-of-the-art VAE-base methods (Vahdat & Kautz 2021 Xiao et al. 2021 Sinha et al. 2021), even when most of these methods utilize powerful hierarchical VAE-based backbones. In contrast, DiffuseVAE utilizes a simple VAE backbone with a very poor baseline FID score and it would be interesting to benchmark LSGM using a simple VAE backbone as ours (some initial evaluations on CIFAR-10 already suggest. that L.SGM might perform much worse than DiffuseVAE with a simple VAE baseline D. In this work our CIFAR-10 10 order of magnitude smaller than LSGM (See Table
model is the same size in (Ho et al., 2020) which 14h. Indeed, like LSGM, DiffuseVAE. can also take advantage of larger model sizes (DifluseVAE-TM with Ex-PDE achieves FID of 2.62 and mean TS of 9.75 on CIFAR-10. See Appendix G.4 Moreover, to the best of our knowledge, DiffuseVAL is the first model to outperform StyleGAN2 (Karras et al., (2020a) on this benchmark while being trained using non-adversarial losses and retaining access to a low-dimensional latent code.
We also benchmarked DiffuseVAE (with Ex-PDE) on two popular face image benchmarks: CelebA-64 and CelebA-HOQ HO-A--QQ-266 256. On the CelebA-64 benchmark, DiffuseVAE performs comparably with the DDPM baseline. Similar to CIFAR10, DiffuseVAF, outperforms other VAF-based methods (Sinha et al.) 2021
Aneja et al.
2020 Xiao et al.
2021 by a significant margin. We observed similar trends on the OclebA-HQ-2555 dataset where DifluseVA VAE outperforms competing VAE based methods except LSGM and is comparable to VQGAN [sser et al. 2020). However, when comparing with LSGM on this benchmark, similar arguments as pointed datasot, samples generated during intermediate training stages (and even after convergence) suffer from color bleeding. We found that this problem can be alleviated by using temperature sampling in the second stage DDPM latents (Appendix G.4.. Therefore, only for T 1000, report the FID scores this benchmark with scaling factor of 0.8.
out for CIFAR-10 hold. Interestingly, we found that for CelebA-HQ-22
1 httppp///.ggeaaeteee..eeaaaaatee

---

Published in Transactions on Machine Learning Research (11/2022)
Method
FID@50k
DiffuseVA (Form-l, Te T= 1000, GMM-75
4.05
Method
FTD
DAE (Form-2, T=1000, GMM=75)
DDPM (T=1000, Our impl.)
3.97
3.93
DifiuseAA (T=1000, GMM=100, FID@10K)
VAE Baseline (GMM 100, FID&H10K
97.07
11.28
VAE Baseline (GMM=76)
72.11
D2C (Sinha
2021
5.7
LSGM (Vahdat et al. 2021
7.22
NCP-VAE [Aneja ef ml 2020
VAEBM IXiso &L
5.31
5.25
VOGAN
D2C (inha et al.
Transtormer Fsseor
2021
2020
18.74
10.2
NVAE (ahdat & Kautz
NCSN Song & Esroon
2021
14.74
25.30
DOVAE Parmar et al 2021
VAEBM Xiño
20.38
15.81
QA-GAN [PARIMAAA & Channa pppayya
NCSNv2 Song & Ermon 20206
2019
10.23
6.42
NVAE Vahddat Kautx
NCP-VAF
Areej At
2020
40.26
24.8
COCO-GAN
4.0
Table 6: Generative performance on CelebA -HQ-2256
Table 5: Generative performance on CelebA-64
4.5 Generalization to different noise types
To test if DiffuseVAE can generalize over different types of noisy conditioning signals during sample gener- ation, we condition the second stage DDPM model in DiffuseVAF. (pre-trained on the CTFAR-10 dataset) different types of noisy conditioning signals (instead of the VAE reconstruction). More specifically, we experiment with two such types of conditioning signals obtained by adding noise to CIFAR-10 test samples: downsampling CIFAR samples to 16x16 resolution (effectively blurring them when scaled back) and adding Gaussian noise (with standard deviation = 0.3). Final DiffuseVAE samples obtained after conditioning on these noisy inputs are visualized in Fig. 7 (with additional results on the CelebA-HQ-Ie samples illustrated in Fig. 16]. We observed that DiffuseVAE is able recover the original samples from the noisy inputs which demonstrates generalization to different noisy conditioning inputs.
Intuitively, these results can be expected since, during training, the proposed DiffuseVAA method learns to refine VAE reconstructions which lack a lot of detail. Hence the task of refining these reconstructions might be more challenging, thus allowing the network to generalize to simpler tasks inherently as illustrated above. However, it is worth noting that certain artifacts in the generated refinements are evident (For instance in Figure 16 the sample quality shows a sharp degradation as more noise added to the conditioning signal), leaving scope for design of more stronger conditioning mechanisms in diffusion models that allow adapt conditional diffusion models on downstream tasks like image super-rosolution in an out-of-tho-box fashion.
5 Related Work
Following the seminal work of (Sohl-Dickstein et al. 2015 Ho et al. 2020 on diffusion models, there has
been lot of recent progress in both unconditional (Nichol & Dhariwal 2021 Dhariwal & Nichol, 2021
Kingma et al.] 2021) and conditional diffusion models (Ho et al. 2021
Saharia al. 2021 Choi et al. 2021
Chen al. 2020) (including scoro-based models (Song et al. 2021b Song & Ermon 2020a for varicty of
downstream tasks including image synthesis, audio synthesis and likelihood estimation among others. Here we only compare DiffuseVAF to recent methods which attempt to combine VAFs with diffusion models. We refer the readers to Appendix C for a detailed comparison of DiffuseVAE to other types of model families.
Among recent advances, there are several works which apply diffusson models in the latent space of powerful autoencoding baselines. D2C (Sinha et al.] 2021) utilizes a learned diffusion-based prior over the NVAE (Vahdat & Kautz, 2021) latent representations while also refining the latent space using a contrastive loss. LSGM Vahddtt et al. 2021) performs score-based generative modeling in the latent space of NVAE baseline. Similarly, Latent Diffusion Models (LDM) (Rombach et al. 2021) apply diffusion models in the latent space of a powerful pretrained VQ-GAN [Esser et al. 2020) autoencoding bascline. In contrast, our method refines "blurry" reconstructions generated by an extrel lightweight VAE using downstream diffusion model. A possible benefit of having a generator-refine framework in contrast to the latent diffusion framework could be the requirement of a powerful VAE bascline as a pre-roquisite to generate high-quality samples. Since there exists a trade-off between latent code disentanglement and high quality reconstructions (Higgins et al. al. 2017). the need of high fidelity autoencoding baseline can be disadvantageous in situations where fine-grained
12

---

Published in Transactions on Machine Learning Research (11/2022)
Noisy Conditioning signal 0.3
Reconstructed Samples
Original Samples
Blurry Conditioning signal (16 16)
Figure 7: Illustration of DiffuseVAE generalization to different noise types in the conditioning signal on the CIFAR-10 test set.
control over the generated samples is required. We hypothesize that this problem is alleviated in DiffuseVAE since our first stage model can readily tradeoff more disentanglement for lower fidelity reconstructions due to powerful second stage diffusion-based refiner model. Tastly, we hypothesize that the latent diffusion framework is complementary to DiffuseVAE since the prior used in our VAE training can be modeled using diffusion model.
[Tuo& & Hu 2021 | present a probabilistic autoencoding framework for point cloud generation via a VAF-like encoder and a diffusion model based decoder. Notably, the most closest to our approach is the concurrent work on DifTAE [Preechakul al., 2022) which uses an end-to-end autoencoding framework for conditioning the diffusion process decoder on the latent code output of an encoder. This equips the diffusion model with a low-dimensional latent space. However, since the model is non-probabilistie, DiffAE relics fitting powerful DDIM density estimator on the latent space of the encoder to enable sampling. Moreover, it's unclear if DifTAR exhibits good sample quality when fitting simple density estimators on the encoder latent space. In contrast, sampling in DiffuseVAE is straightforward due to a probabilistic formulation. Additionally, DiffuseVAE can also take advantage of fitting external density estimators on the latent space as demonstrated in this work.
6 Limitations and Discussion
In this work, we presented a novel unifying framework for training VAFs and diffusion models and demon- strated its offectiveness in generating high-quality samples, providing better sample quality vs number of steps trade-off while equipping DDPM with low dimensional latent code which can be used for controllable synthesis using DDPM, and generalizing to different types of noise in the conditioning signal. However, the DiffuscVAE model is not without its limitations:
Due to a generator-refiner framework, the semantics the final generated samples depends largely on the coarse sample generated by the generator model (a simple VAE in our case). Therefore, if the coarse sample is not semantically meaningful, this will propagate the final generated sample after refinement. This can be expected from VAËs due to mismatch between the aggregated posterior g(2) and the prior p(z) during VAE training which we alleviate using Ex-PDE estimation but the problem still persists (which is evident from the gap in sample quality between an unconditional DDPM bascline and DiffuseVA even after Ex-PDE).
2. We also observed that when the conditioning signal provided by the first stage VAE is uninformative (too blurry), the second stage DDPM model can generate unpredictable refinements. On this note, it would be interesting to explore the impact of the choice of VAE on the overall sample quality of the model. Moreover, since we work with vanilla VAEs, some artifacts in controllable synthesis results are evident due to correlated attributo-spccific latent directions (See Figure 15). Using variants like 3-VAEs (Higgins et al. (017)) can help achieve more disentanglement between image attributes leading to better coltrol synthesis results.
13

---

Published in Transactions on Machine Learning Research (11/2022)
3. In this work, since we focus sample quality, did not explore the impact of the diffusion model training on the latent space of the VAE when trained end-to-end. It would be interesting to explore if end-to-end training might alleviate some problems with VAE's.
Lastly, would interesting to explore stronger conditioning mechanisms in the context of diffusion models which reduce the reliance of the final sample on the stochastic DDPM sub-code. In the context of DiffuseVAF, this also be useful in improving model generalization to downstream tasks like image super-rest super-resolution and denoising presented in Section 4.5
Broader Impact Statement
In addition to modelling images, our proposed approach can also be used to model data of other modalities like speech, text, etc. Tt has the potential mitigate bias and privacy issues for related MT. models that require data collection and annotation. However, such techniques could also be misused produce fake mislcading information, and rescarchers should be aware of these risks and explore the proposed approaches responsibly.
Acknowledgments
We would like to thank Ben Poole for his insightful comments and suggestions through the course of this project. We would also like to thank Google Cloud for supporting our research in the form of cloud compute credits.
References
Jyoti Aneja, Alexander G. Schwing, Jan Kautz, and Arash Vahdat. Nep-vac: Variational autoencoders with noise contrastive priors. ArXiv, als/2200.00291,, 2020.
M. Bauer and A. Mnih. Resampled priors for variational autoencoders. In Proceedings of the 22nd In- ternational Conference Artificial Intelligence and Statistics (AISTATS), volume 89 of Proceedings of Machine Learning Research, 66-75. PMLR, April 2019. URL http://proceedinng: mlr press/v89/
Yuri Burda, Roger Grossc, and Ruslan Salakhutdinov. Importance weighted autocncoders, 2016.
Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 8-vae, 2018.
Bing Cao, Han Zhang, Nannan Wang, Xinbo Gao, and Dinggang Shen. Aulo-gan: self-supervised collabora- tive learning for medical image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 10486-10493, 2020.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan secretly an ensrgy-based model and you should discriminator driven latent sampling, 2021.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation, 2020.
Ricky T. Q. Chen, Xucchen Li, Roger Grosse, and David Duvenand. Isolating sources of disentanglement variational autoencoders, 2019.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images, 2021.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilv: Conditioning method for denoising diffusion probabilistic models, 2021.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXie preprint arX3: 2019.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans image synthesis, 2021.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks, 2016.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models, 2020.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020. URL https://arxiv..rrrt org/ wttp://.rr//bb////////// 2012 09841
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. ArXio, abs/1903. 199/1993.1246,, 2020.
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free- form continuous dynamics for scalable reversible generative models, 2018.
Martin Hcusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium, 2018.
I. Higgins, L. Matthey, A. Pal, Christopher P. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint wrXiv:2106.15280 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
Tero Karras, Mika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, 2020a. URL https://arxiv.org http:////g///////////e/eee 2006 00606606666666
Tero Karras, Samuli Laine, Miika Ailtala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2020b.
Diederik P. Kingma and Prafulla Dhariwal. NewrIPS, 2018.
Glow: Generative flow with invertible Ixl convolutions. In
Diederik P Kingma and Max Welling. Auto-cncoding variational bayes, 2014.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learning with deep generative models, 2014.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling, Improving variational inference with inverse autoregressive flow, 2017.
Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32 33, 2009. 7/www.co cs. toronto.
URL https:
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. Tn IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2020a.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Wonkwang Lee, Donggyun Kim, Seunghoon Hong, and Honglak Lee. High-fidelity synthesis with disentan- gled representation, 2020b.
Chieh Hubert Lin, Chia-Che Chang, Yu-Sheng Chen, Da-Cheng Juan, Wei Wei, and Hwann-Tzong Chen. Coco-gan: Generation by parts via conditional coordinating, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Eric Luhman and Troy Luhman. Knowlodge distillation itcrative generative models for improved sampling speed, 2021.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation, 2021.
Vaden Masrani, Tuan Anh Lc, and Frank Wood. The thermodynamic variational objective, 2021.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks, 2018.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021.
Erik Nijkamp, Mitch Hill, Song- Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcme toward mergy-based model, 2019.
Anton Obukhov, Maximilian Seitzer, Po-Wei Wu, Semen Zhydenko, Jonathan Kyl, and Elvis Yu-Jing Lin. High-fidelity performance metrics for generative models in pytorch, 2020. URL https://github. com/
feshassttorch-fidell1 Version: 0.3.0, DOI:
KANCHARLA PARIMALA and Sumohana Channappayya. Quality aware generative adversarial networks. H. Wallach, H. Larochelle, A. Beygelzimer, F. Alche-Buc, E. Fox, and R. Garnett (eds.), Advances Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URT. https:// .pd
proceedings httpp////////reeaaaaaaaa02222220
Gaurav Parmar, Dacheng Li, Kwonjoon Lec, and Zhuowen Tu. Dual contradistinctive generative autoen- coder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 823- 832, 2021.
Adrian Alan Pol, Victor Berger, Gianluca Cerminara, Cecile Germain, and Maurizio Pierini. Anomaly detection with conditional variational autoencoders, 2020.
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In IEEE Conference on Computer Vision and Pattern Recognátion (CVPR), 2022.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprinl arXiv:2104.07636, 2021.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations, 2022. URL http:p////ggeneeeeeeeee////o///a
16

---

Published in Transactions on Machine Learning Research (11/2022)
Tim Salimans, Ian J. Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NIPS, 2016.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-denoisingg models for few-shot conditional generation. arxiv preprint urXi::2166.06819 2021.
Samarth Sinha and Adji B. Dieng. Consistency regularization for variational auto-encoders, 2021.
Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. arXiv preprint or 5i::2104006554 2021.
Jascha Sohl-Dickstein, Fric A. Weiss, Niru Maheswaranathan, and Surya. Ganguli. Deep unsupervised learning using nonequilibriu thermodynamics, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2021a
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, 2020a.
Yang Song and Stefano Ermon. Improved techniques for training score- based generative models, 2020b.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stelano Ermon, and Ben Poole. based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b. URL
Casper Kaae Sonderby, Tapani Raíko, Lars Maaløe, Søren Kaac Sønderby, and Ole Winther. Ladder varia- tional autoencoders, 2016.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder, 2021.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space, 2021.
Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling. Sylvester normalizing flows for variational inference, 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. -309963188, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860961.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sample from diffusion probabilistic models, 2021.
Yuxin Wu and Kaiming He. Group normalization, 2018 URL httpp://arriv.orr/aabb///e 08494
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models, 2021.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemmaa with de- noising diffusion GANs. In International Conference on Learning Representations, 2022. URL https:
17

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 8: Forward Process
Figure 9: Reverse Process
A
Background on Diffusion models
2015 Ho et al. 2020 are latent-variable models consisting of a forward noising process (g(01.7 Tol) [orresponniig to an inference model in other generative model families like VAFs (Kingma & Welling 2014 Rezende & Mohamed 2016). See Fig. S and a reverse denoising process (p(zo:r) (corresponding to a generator or decoder in VAEs. See Fig. 9h. The forward process is modeled using& a Markov chain which gradually destroys the structure of the data vp over a number of time-steps T. Similarly, the reverse process also modeled as Markov chain which learns recover the original data zo from the noisy input zy. The form of the forward process and some notable properties of the forward process conditional distributions are summarized in the following equations Eqs. 13 19
DDPMs (Sohl-Dickstein et al.
T
((iT)/=)) = II ((
(13)
q(<r|==-1) = = 3, B,1)
(14)
The forward process of DDPMs admits a closed form for xy for any t, as follows:
(1 T))
(15)
where az (1 3x) and ō II on
(16)
The forward process posteriors are also tractable and are given by
(51-11221, NN (0),
(17)
where 20) Vāl-
var(l - aa;1)
2o +
24
(18)
aq
1 1 Cq.
0tt- and B,
31
(19)
1 āi
The reverse process also be parameterized using first-order Markov chain with learned Gaussian transition distribution follows
T
p(Zor)) = p(xT) II
po (x
(20)
Po(x;- 1xx)) = N(Ho(a t), Eg(x,,
(21)
po( (Npo(( t), (o(r),t))
(22)
Given large enough and well-behaved variance schodule of 3. the distribution g(zT(2)) will approximato an isotropic Gaussian. We can generate new sample from the underlying data distribution q(zg) by sampling latent from p(Er) (chosen to be isotropic Gaussian distribution) and running the reverse process. As
18

---

Published in Transactions on Machine Learning Research (11/2022)
proposed in (Ho al. 2020), the reverse process in DDPM is trained to minimize the following upper bound over the negative log- likelihood (See (Sohl-Dickstein et al. 2015) for detailed proofs):
B D DxL (kol((or((o))lltor))))
Σ
- log po(ooa))))
(23)
A notable aspect of the above objective is that all the KT. divergences involve Gaussians and, consequently, are available in closed form. Notably, (Ho et al. 2020) parameterize the reverse process conditional pa pe(21-1
using the forward process posterior q 21, To). (Ho et al. 2020) show that such a parameterization
simplifies the second term Eq. 23 at any given time-step to the lollowing objective Eq. 24
co-aaoo + a,c
(24)
where Voszo + <vT-ā, and ~ NN, T). Intuitively, this means that the reverse process in DDPM is trained to predict the noise added to the input zo at any time-step t. We use this simplified training formulation throughout our work to train all proposed parameterizations of diffusion models as (Ho et al. 2020) show that this formulation yields superior sample quality than other forms of reverse process param- eterizations. For further details on the exact training and inference processes, we encourage the readers refer to (Ho et al. 2020).
19

---

Published in Transactions on Machine Learning Research (11/2022)
B Discussion of DiffuseVAE (Formulation-2)
Algorithm DDPM Training (Form. 2)
Algorithm 2 DDPM Inference (Form. 2)
repeat
Evan NN T)
((r))
y VAEDEC(Zva) (Zvae)
80 VAE(2a)
2T NYy,1) for ' to do
~ Uniform((1.... ... T})
6 I),,
2 = NN(,, f), if > else
Take gradient descent step on:
8o - (#t
c y, 1) ai)
va,
Vale Eø( 2 to vI 0re, I, f0)l
4 8o + Tt + Y21
until convergence
Tt-1 f + 20;
end for
return To y
The DDPM training objective proposed in (Ho et al. 2020 has the following form:
Bg
DxL (1((7)=)))((((7)) +
Σ Dxi(g(a, 2o) 110 log po ( 21)
(25)
Lo
B.1 Reverse Process parameterization
Following Ho et al. 2020), paramoterize the reverse process transition (= using the functional form of the forward process posterior 4(<2-1/=4 x.)) For the DiffuseVAE formulation proposed in Section 3.4.2 in our paper, the forward process conditional distributions can be specified as:
((1/)1(-1,20) . (v1 (V1--But- BEIC-1 + (1 V1- V1-B,)00 311) where t>
(26)
( 20) N (Varza + 20, (1 - a.)1)
(27)
The posterior distribution g(xx-1 [r, No, to) will also be a Gaussian distribution with the following form:
Fu. 20) = (Ae(Ie, Fo. 2o). BrI)
(28)
where,
3r 3.yar-1
(1 ār-
1)
ā,-1) t0
(0,
D0 +
Da + (1
(29)
1 σ
1 σe
B (1 az-1)
1
3, and vo
(5r S0 -evl- σt
ag where N)(,
Var
(30)
Hence the forward process posterior in this DiffuseVAE. formulation is a shifted version of the forward process posterior proposed in (Ho et al. 2020). Since the VAE reconstruction do for an image To is constant during DDPM training, we can parameterize the reverse process postcrior as (oo $o t) Ao vg. + kōgg Additionally, we keep the variance the reverse process conditional fixed and equal to 3 as proposed in (Ho al. (020). Since I,-1 x (I,, Wo y) f (v,, To, y. t) the DDPM training objective our formu- lation remains unchanged from the simplified denoising score matching objective proposed in (Ho et al. 2020)
20

---

Published in Transactions on Machine Learning Research (11/2022)
B.2 Choice of the decoder, Lo
One possible choice for the decoder is set po( (21) to be discrete independent decoder derived from the 1, we have foo &., 1) = zo(21, &o. fo)-
Gaussian (21 &o, 1), 32 (Ho al. 2020). However, Therefore, to account for the VAE reconstruction bias in the final DDPM output, we set our decoder 22(2)(21) = (1 &o, 1) &o, 311). Without using this adjustment, we found the final DDPM samples to be a bit blurry in our initial experiments. The final training and inference algorithms are summarized in Algorithms 1 and 2 respectively. In Algorithm 2
the coefficients Yo. and "2 denote the coefficients of the
forward process posterior in Equ. 29
B.3 Integration with DDIM
We now derive the updates for the DiffuseVAE formulation-2 when combined with DDIM sampling. Given the form of the forward process marginal as in Equ. 27 we assume the following form of the forward process posterior:
q De, 20) 2o) (Ne,
(31)
Pt V 1
V ac-1 - of
Ft
k20
(32)
C2
o7
C4
Ct
(33)
1
04-1
We now have,
q( 120, 20)
90 vo. f0)g(2, (2 fo)dr
(34)
Since both the distributions within the integral are gaussians, the resulting marginal will also be gaussian with the following form:
@(xr-1/20, 20) = -N oi)
(35)
K v1 04 oi
7
V 04- 130
80
(36)
a2
// = 1 - Ō4.1
(37)
However, already know the form of the marginal g( Fog x00 20) from Eqn. 27
follows:
q 20) =
+ so. 6(-1
(38)
Therefore implies that,
01-1 11 o
K
(39)
This completes the analysis of the modified DDIM forward process posterior which is compatible with DiffuseVAE formualation-2
B.4 Primary Intuition
The primary intuition behind constructing such a formulation is that by initializing the base distribution from VAE reconstruction, we can hope to speed up the reverse diffusion process. In the low time-stop regime, DiffuseVAE (Form-2) usually performs better than (Form-1) (See Tables 3 11 and 12 These results indicate hypothesis might hold valid in the low-time-step regime in diffusion models.
21

---

Published in Transactions on Machine Learning Research (11/2022)
C
Related Work
Recent work in MSPMs also includes improving the speed vs sample quality tradcoff in the DDPM sampling
process [Song et al. 2021a Watson et al. 2021 Luhman & Luhman 2021 Salimans & Ho 2022 X Xiao et al. 2022 We consider these advances in speeding up diffuuion models are complementary to our work and can also be used to improve the sampling efficiency of DiffuseVAE. However, on the contrary, a majority of such methods were designed for improving sampling spoeds in DDPMs while DiffuseVAE, improves this tradcoff inherently. Similarly for VA (Kingma & Welling) 2014
Rezende & Mohamed 2016), there has also been
progress in improving the ELBO estimates (Sinha & Dieng 2021 Burda et al., 2016 Masrani et al.) 2021) and image synthesis (Child 2021 Vahdat & Kautz] 2021 Lee et al.
2020b; Xiao et al. 2021). Next, we
compare our proposed approach in detail with several of these related existing model families.
Unconditional DDPM: DDPM/DDIM as introduced in (Ho et al. 2020 Song et al. 2021a) lacks a low-
dimensional latent code which limits model application scope in several downstream tasks. In contrast, DiffuseVAE equips diffusion models with a low dimensional latent code that can be utilized for downstream tasks including but not limited to controllable synthesis. Moreover, we demonstrate better speed quality tradcoff in DiffuseAAE as compared to standard unconditional DDPM/DDIM models and that the conditioning signal in DiffuseVAR helps in generalization to noisy conditioning signals.
Conditional DDPM: Conditional DDPM as introduced in (Ho et al. 2021 and (Saharia et al. 2021 uses a cascade of multiple diffusson models (CDMs) for generating high-resolution images. However, for even a two-stage pipeline, the sampling time of such models would be effectively much higher than DiffuseVAR. Given the flexibility of our approach, we hypothesize that a single-stage VAE can also be replaced by a complex multi-stage VAE architecture proposed in |Child 2021 Vahdat & Kautz, 2021 for comparable sample quality to cascaded diffusion models without affecting the sampling time significantly. Moreover, such cascades lack a low-dimensional latent code which might be limiting factor for certain downstream applications. It is worth noting that, (Ho et al. 2021) use a conditioning augmentation scheme where the reslutosnn image is generated by conditioning on a blurred/ low resolution image. In contrast, our model is already conditioned on reconstruction generated by a VAE (which is inherently blurry) and some sense resembles the houristic employed in CDMs.
VAE based methods Hierarchical VAEs (Sendebyy et al. 2016 Vahdat & Kautz 2021
Child 2021:
Razavi et al., 2019) can suffer from posterior collapse and heuristies like gradient skipping and spectral normalization (Miyato et al. 2018) might be required to stabilize training. Morcover, these models require a large dimensionality of the latent codes to generate high-fidelity samples (Vahdal & Kautz 2021 Razavi et al. 2019). In contrast, DiffuseVAE training does not suffer from such instabilities and provides access to a single latent code layer (with dimensionality comparable to GANs) to generate high-fidelity samples. Among other recent works, VAEBM (Xiao et al. (021) uses EBMs (Du & Mordatch 20200] Nijkamp et al.,
2019) to refine VAE samples while LSGM (Vahdat & Kautz, 2021 perform score-based modeling in the
latent space of VAE backbone. However, both VAEBM and LSGM use NVAE ( (Vahdat & Kautz |021] as the base VAE architecture which also lacks low-dimensional latent code. (Lee et al. 2020b) distill the disentanglement properties in the VAE latent code to the latent space of a GAN-based generator. However, this approach would also suffer from existing problems of training stability and mode-collapse in GAN-basod models. On the other hand, DiffuseVAE does not suffer from such problems
22

---

Published in Transactions on Machine Learning Research (11/2022)
D Detailed Proofs
D.1 Derivation of the DiffuseVAE objective
Given a high-resolution image an auxiliary conditioning signal to be modelled using a VAE, a latent representation 2 associated with y, and a sequence T representations Fı.T learned by diffusion model, the DiffuseVAR, generative process, p(Zorr, y; 2) can be factorized as follows:
p v, = p(e)po((ll)ppoccrrll.
(40)
where 0 and & are the parameters of the VAE decoder and the reverse process of the conditional diffusion model, respectively. The likeliood of the training data can then obtained as:
log v) log / p(FO:T: : y, 2)d21.d2
(41)
Furthermore, since the joint posterior p(a1.r = y, 2o) is intractable to compute, we approximate using a surrogate posterior (( mo) which can also be factorized into the following conditional distributions:
( zly, mn) (x (1)
(42)
where di are the parameters of the VAE recognition network (go (2ly, vo)) Since computation of the likelihood in Eq- 441) intractable, we can approximate by computing a lower bound (ELBO) with respect to the joint posterior over the unknowns (21.7, 2) as:
p(Zorr, y, 2) log logp(20.2) y) 2 (e(oor)) log g( 2 2/20, y)
(43)
Plugging the factorial forms of the DiffuseVAE generative process and the joint posterior defined above in eqn. 43 we can simplify the ELBO as follows:
log p(rorr 2)
log p(r,, v) p alusao)
(44)
g 2 y. 2o)
p(z) (xo log go (z y, 2o 2 (1( U, 2, xo)
(45)
p(=)
Do (x ly, z)
log qo (2 v0)
logro(y(()) + log
(46)
(x Vg 2, mo)
log
p(2) logpo(Ulz)
+ E (g(oa
Po ( 1/) 2) log (x (y) 2, xo)
(47)
(2 zn)
Po(coorly,
E (e [po(y|=)] - DKrllao + +Beng(ely.eo)
E Ey(zurlb.e,
g (y, v0)
EVAR
CDDPM
(48)
D.2
Derivation of the DiffuseVAE (Formulation-2) marginals
Given:
q(11|20,00) = N 3120 + 2o. 311)
(49)
(1 20) = NN Byxz-1 + 111-B.)oo.. 3r1)
(50)
From Eqn. 50
can write,
St /1 11.-0.0--00 + 1--------- Bi) + e VB where e N((,T)
51
23

---

Published in Transactions on Machine Learning Research (11/2022)
Taking expectations both sides,
E(x,) / -
8,) ((xx) /1- 3 [1 (1- (1 (1-v1- -1- 101-3--0 32-1)00 (11 v1- 3,)
(52)
B(z)) = v(1 -1--111 B.)(1 -(- (1 ((1- :30(1 Bc-1) 80
(53)
E(x:)= I (.))(a)) + 20
1
11 3e)
(54)
Substituting [E(1)) vI 1 + 20 from Equ. 49
into the above formulation we get,
[(x;) =
II
30 + fn = v Vāgan + So
(55)
Similarly can be shown that Var(x;) (1 ã.)l. Therefore,
q( 2o) NVVar + to. (1 az)
(56)
24

---

Published in Transactions on Machine Learning Research (11/2022)
Method
FID@10K +
Method
FID@10k
DiffuseVAE (fo)
5.94
DiffuseVAE (Two-stage)
6.81
DiffuseVAE (ău + Latent code)
6.07
DiffuseVAB (End-to-end)
8.12
Table 7: FID (10k samples) comparison between dif-
Table 8: FID (10k samples) comparison between two- stage and end-to-end (raining on CIFARIO.
ferent DiffuseVAB conditioning schernes on CIFAR1O.
E
Justification of the design choices in DiffuseVAE
Here we justify the design choices made in the DiffuseVAE model specification.
1. Choice of the conditioning signal y: The choice of assuming the conditioning signal in Eq. 10 to be the training data To is motivated by the task of refining the blurry samples generated by a simple VAE model using a DDPM model.
2. Choice of the conditioning signal z: The choice of conditioning the DDPM model on ån (the VAE reconstruction of the training data zo) instead of the VAE inferred latent code 2 (usually lower- dimensional) allows us to condition the second stage DDPM directly on samples drawn from another model (not necessarily VAE) or on real images, which can be quite useful as illustrated in Section Additionally, there can a variant of our method in which the DDPM model conditioned on both 2 and é. We conditioned the DDPM decoder on 2 using Adaptive group normalization layers
4.5
DDhriiwal & Nichol 2021 Wu & He 2018) as follows:
y MLP(=) + ez
(57)
AdaGN(h, (h, y) y, GroupNorm(h)) + yb
(58)
where h is the output of the first convolution in the residual block and y [ys, V6] is obtained from the latent code and the time-step embedding e,. On benchmarking this DiffuseVAB (Formulation- 1) variant on CIFAR-10 trained for around 1.1M steps, we found that the resulting model exhibited slightly worse performance compared to the DiffuseVAR. variant conditioned only on the VAF, recon- structions (fo) (See Table 7) Therefore, we only condition the DDPM model in DiffuseVAE only on the VAE generated reconstructionss.
3. Two-stage training: The choice of a two-stage training approach in DiffuseVAE is motivated by two reasons. Firstly, in our early experiments on CIFAR-10, we observed that the end-to-end model exhibited much worse performance than its two-stage counterpart during inference (See Table g where both models were trained for 400k steps. Secondly, from a computational standpoint, using a two-stage training formulation would be more amenable training on limited compute resources end-to-end training would require both models to fit. in memory.
25

---

Published in Transactions on Machine Learning Research (11/2022)
CIFAR-10
CelebA-04
CelebA-HQ-128
CokbA-HQ-220
LHQ-266
Stage-1 Hyperparameters
Resolation
32 x 32 [0. 1]
64 x 64 [0. 1]
128 x 128
258 x 206
125 x 12s
Data
Data Range
10, 3]
[0, :]
[0, X]
Model
parameterr
9.2M
21.1M
32.732
36. 36.3M
Random Seed
0 No
0 No
0
0 No
0 No 206
Mixed Precision
No
Effective Butch Sisc
128
128
128
82
Training
Optimuxer Latent code size Kl.-weight
of spocha
Adam( 1n-4)
500
Adam Adam (Irm 1m-4)
250
Adam( 1o-41
500
(damm(em1e-4)
50d
Adamm
500
612
1024
1024
GMM)N=77)
GMM(N=100 100)
GMMNN=100
GN=M(N=100)
Stage-11 DDPM Hyperparameters
Resolation Flip
64 x
128 128
x 200
250 26G Yee
Yae
Yos
Range
|+,, 1]
[+L, 1]
[-1. 1]
[-1,
a of channels
126
126
1225
125
125
Seniete) of atsention block # of atsention heads
[10]
[10]
[L0]
[10]
[10.8]
8
# reaidual blocke per scale
Model
Channel multiplure
(1,2.222)
(1:2,2/2,4)
(1,2/2,3.4)
(1.1.2.2,4.4)
(1.1.2,2,4.4)
# of parameters
35.7M
84.6Ml
95.20
113M
114M
Dropout
0.3
0.1
0.1
(defauls)
Linear(1edd, 0.03)
Linex((1e-d; 0.02)
Linear( 0.03) 1000
Lines[[eed 0.02)
Eieaa((eed, 0.02)
time-stepa (T)
1000
1000
1000
1000
Random rand Mixed Pracision EMA decay vate Effective baich sise
0
0
0
0
0
No 0.9000
No
No
No
No
0,9800
0,9009
0.0000
0,9900
128
1.28
64
64
64
Training
# of atepa
1.1M
0.54b
0.30M
0.36M
Optimiser
Adaan(I- 2o-4)
(la
Adamm(]e 20-5)
Adamm(e-2ee 22-5)
Adiam:(ee22es. (lem2
Grud, Clip Thesheld steps
1.0
1.0
type
Natee prediction (L2)
prodiction
Naiee predsction (L2)
prediction (1.2)
prediction (12)
Evaluation
Variance
Axedlarge
fizedlarge
Sxeddmall
Axxedmall
fizeddmall
Table 9: Hyperparameters for the training setup in DifTuseVAE
F Training and Hyperparameter details
All hyperparameters details related to VAE and DDPM training in DiffuseVAE are listed in Table g Moreover, all hyperparameters (model and training) were shared between both DiffuseVAE formulations.
Data preprocessing: During the first stage VAE training, all training data was normalized be- tween [0.0, 1.0]. For the second stage DDPM training, the training data was scaled between [-1.0, 1.0] (including unconditional baselines and DiA formulations). We also applied random horizontal flips as form of data augmentation to the training images during the second stage DDPM training
Model architecture: We use the same network architectures as explored in prior work in diffusion models (Ho et al.] 20200 Dhariwal & Nichol, 2021 Nichol & Dhariwal (202). The VAE architecture used for Stago-1 training consists of residual block architectures inspired from CCiid 2021 (Refer to our code for exact architectural details). The VAE latent code size was set to 1024 for LHQ-256 and CelebA-HQ (both 128 and 256 resolution variants) and 512 for the CIFAR-10 and CelebA (64 x 64) datasets. We do not investigate the effect of the size of the latent code in this work. Similar to prior work (Ho et al. 2020), for all datasets except CIFAR-10 models used in SoTA comparisons, we use the U-Net (Ronneberger et al. 2015) decoder implementation from (Nichol & Dhariwal 2021 in the reverse process Stage-If DDPM training. For the CIFAR-10 dataset, we used the U-Net decoder implementation from DDIM (Song et al., 2021a) (httpa://github. com wtt./oom//o//eeeeeee hwtp://w/wppdmmooooo/ooooeeeeeee htt.:nngggon//daaaa///////////aa http:///////ii////o//////a/aaaaa pyy. The U-Net decoder model hyperparameters are listed in Table g
Training and Inference: Unless specified otherwise, we use the same hyperparameters during training as proposed in (Ho et al. 2020). All DDPM models were trained using the simplified objective proposed in (Ho et al. 2020). We used a mix of Nvidia 1080Ti GPUs (44GB memory), a cloud TPUv2-8 (64GB memory) and cloud TPUv3-8 (128GB memory) for training the models. Specifically, we used the GPU setup for training our CIFAR-10 and CelebA-64 models while we utilized the TPUv2-8 for training CelebA-HQ
26

---

Published in Transactions on Machine Learning Research (11/2022)
models at the 128 128 resolutions. Finally, we utilized the TPUv3-8 model for training CelebA-HQ and LHQ models at 256 256 resolution.
Evaluation: For FID Heusel et al. 2018 score computation, we utilized 10k samples for the CelebA- HQ-128 dataset and 50k samples for state-of-the-art comparisons on the CIFAR-10 and the CelebA-64 datasets. For CelebA-HI 256 comparisons we computed FID scores on 30k samples since the CelebA-HQ dataset contains 30k images. We used the torch-fidelity (Obukhov et al. 2020) package for FID and IS score computations. In this work, when saving samples to disk, used standard denormalization (ie. 0.5 img + 0.5) for all datascts. We used our GPU setup primarily for evaluation.
27

---

Published in Transactions on Machine Learning Research (11/2022)
CIFAR-10 (FID@50k)
CelebA-64 (FIDa50k)
CelebA-HQ-256
(FID&10k)
Baseline VAE DiffuseVAR (Form-1, Ex-PDR)
137.68 2.83
72.11 4.05
97.07
11.28
Table 10: Quantitative comparison between sample quality first stage VAEs in DiffuseVAE (Generator) and the final DiffuseVAF. samples (Refiner)
CelebA-61
CIFAR-10
10
25
50
100
10
25
50
100
DDPM (uncond)
37.31
17.06
10.99
8.26
42.66
15.97
9.98
7.76
DiffuseVAE (Form-1, Ex-PDE)
26.09
14.16
9.58
7.54
34.19
DifluseAA (Porm-2, Ex-PDE)
25.79
13.89
9.09
7.15
34.22
17.36
11.00
8.28
Table 11: Speed quality tradeoff comparison between DDPM and DiffuseVAE for the CIFAR-10 and CelebA-64 datasets. FID reported using 10k samples
G Additional Results
G.1 Canerator-Refiner Framework
Some additional qualitative results demonstrating the generator-refine framework VAEs are shown in
Fig. 13 Table T0 further supports our qualitative results for several other benchmarks by comparing the
FID scores between Stage-1 VAE generated samples and the corresponding final DiffuseVAB samples.
G.2 Controllable synthesis
The directions for meaningful concepts (or image attributes like gender, age, hair style) are obtained by considering pairs of attribute negative and positive training samples. For each such pair, we compute the latent code representation for the positive and the negative sample and compute the difference between the attribute positive and the negative latent. We repeat this procedure for all such pairs and compute the average of the difference between the latent codes to obtain the direction vector for the attribute. Formally, given an attribute of interest a and the set tuples (I xneg) = of attribute positive and negative images, the latent direction &a given by:
(i)
1 N
2q
Σ
f(a(fl)) f(x(1)g)
(59)
where f denotes mapping from the image to the latent space (the VAE encoder in this case). Given this latent direction, manipulate attribute negative image by simply adding this vector to the latent code representation of the attribute negative image and decoding the resulting latent code representation follows:
2p + Aāa
(60)
where 2, is the latent code representation of the atribute negative image, 2, is the latent code containing the missing attribute and À is scalar which controls the coarseness of the controllable generation (higher values usually result in more coarse generations). In this work, we use the attribute annotations provided by the GelebAMask-HHo dataset. (Lee et al.) 2020a) and a value of N-100 to construct the set of positive and negative samples for any attribute of interest. Additional controllable synthesis (including single attribute manipulation and composite manipulations) results for the CelebA-HQ dataset at the 128 128 resolution are shown Fig. 14 Figure 15 compares between composite odit-based samples generated from our first stage VAE and the corresponding refined samples generated from DiffuseVAF.
28

---

Published in Transactions on Machine Learning Research (11/2022)
10
25
100
DDIM (uncond)
15.19
8,00
6.76
6.24
DiffuseAAE (DDIM, Forml, Ex-PDE)
11.79
7.44
6.51
6.14
DiffuseVAR (DDIM, Form2, Ex-PDE)
12.15
7.63
6.62
6.22
Table 12: Speed vs quality tradeoff comparison between DDIM and DiffuseVAE for the CIFAR-10 dataset. FTD reported using 10k samples
10
25
50
100
DiffuseVAE (DDIM, Form-1)
26.07
19.75
18.90
18.85
DiffuseVA (DDIM, Form-1, GMM=100)
24.09
17.47
16.65
16.63
Table 13: FID scores (on 10k samples) for DiffuseVAE (Form-1) using DDIM sampling for the CelebA-HQ- 256 dataset
G.3 Speed vs quality tradeoffs
Reverse Process subsequence selection: We use the linear and quadratic time-step selection as discussed in DDIM [Song el al., 2021a), when running the reverse process for only a subsample of the time-steps for efficient sampling. We call this spaced sampling. For benchmarking both DiffuseVAF. and the baseline DDPM/DDIM models for the speed vs quality tradeoff, we selected the scheme which yielded lower FID values. Hence, we use the quadratic time-step schedule for all datasets when benchmarking DiffuseVAE while used the quadratic schedule for the CIFAR-10 and the CelebA-64 datasets and linear schedule for the CelebA-HQ dataset when benchmarking the baseline DDPM/DDIM. There is also possibility of using bruncoled sampling in which only the last / time-steps are used for sampling. However, we found that the latter yielded inferior results than spaced sampling, so we do not report the FID scores for truncated sampling here.
Additional results on speed vs quality tradeoff: Table shows a speed vs quality tradeoff comparison between DiffuseVAE (with Ex-PDE) and the DDPM baseline for the CIFAR-10 and the CelebA-64 bench- marks. Both methods use the fizedsmall variance type discussed in (Ho et al.) 2020). On the C-4cbA-64 dataset, DiffuseVAE again provides much better speed vs quality tradeoff than standard DDPM. How- ever, on the CIFAR10 dataset, DiffuseVAR lags behind the standard DDPM (except at T=10) in terms of FID scores. This is surprising, since for T=1000, our DiffuseVAA model outperforms our baseline DDPM. However, when using DDIM sampling, DiffuscVAE outperforms the unconditional DDIM (See Table 12 For completeness, we also report the FID scores on 10k samples for our CelebA-HQ-255 DiffuseVAF. (Form-1) model using DDIM sampling in Table 13
G.4 State- of- the-art Comparisons
Model size and Runtime comparison: LSGM and DiffuseVAE. Here we compare model sizes between
DiffuseVAA and LSGM (Vahdat et al. 2021h. h. Table 14 compares the model sizes between the LSGM and DiffuseVAE models on the CIFAR-10 and the CelebA-HQ-2256 benchmarks. LSGM utilizes an order of magnitude larger VAF. backbones and denoising decoders comparison to DiffuseVAF. When computing the LSGM model size, we compute the size of the best FID model (Sce hhttps://githuu httt::::/ggtuu....o/////aaa////s com/NVLabs/LLGG) To examine the performance gains when using larger models, we trained DiffuseVAA (Form-1) model with unchanged VAE baseline but with larger DDPM decoder with around 73M parameters CIFAR-10. Indeed, when using larger model, DiffuseVAR, with Ex-PDE achieves FID of 2.62 and mean IS of 9.75 on CIFAR-10 which shows that our model can take advantage of larger model sizes as well.
We further benchmarked DiffuseVAE and LSGM CIFAR-10 models terms of the wall-clock time and memory required for sample generation on a batch size of 64 samples on a single Nvidia 1080Ti GPU. In terms of memory consumption, the LSGM model consumes 5.1GB in comparison to around 2.00GB consumed by DiffuseVAE. This is to be expected due to larger LSGM model size. Interestingly, L.SGM
29

---

Published in Transactions on Machine Learning Research (11/2022)
CIFAR-10
CelebA-HQ-220
LSGM (VAE backbone)
86,6M
50.9M
LSGM (Denolsing decoder)
375.6M
408. 408.4M
DifluseVA (VAF, backbone)
9.2M
32.7M
DiffuseVAR (Denaising decoder)
35.7M
113.3M
Table 14: Model size comparison (in terms of the number of paramcters) between DiffuseVAE and LSGM on the CIFAR-10 and CelebA-HQQ2255 benchmark
(using 140 NFEs) only takes 67.03s to generate a batch of 64 samples as compared around 103.13s required by DiffuseVAE (using 1000 NFEs). We hypothesize that this gain is primarily due the efficacy of applying diffusion in the latent space in LSGM compared to the pixel-space in DiffuseVAF. However, this design choice also prevents access to compact latent space in LSGM.
G.5 Temperature Sampling DiffuseVAE
We experiment with a temperature scaling technique where during the DDPM sampling stage in Diffu- seVAF, we sample the initial DDPM latent gy from a base Gaussian distribution with standard deviation scaled by A. This common technique utilized in prior works (Vahdat & Kautz 2021 Kingma & Dhari- wal. 2018) to tradeoff between sample quality and diversity. Interestingly, dalaset, samples generated during intermediate training stages (and even after convergence) sulfer from color bleeding as shown in Fig. I0 (Top Row). We found that by applying temperature annealing in the second stage DDPM latents alleviates this problem (See Fig. Bottot Row)). Therefore, we compute FID for state-of-the-art the-art comparisons on this benchmark with a scaling factor 0.8. We did not observe such color channel bleeding samples of other benchmarks. In such cases, we observed that temperature scaling did not help and thus was not used report FID scores.
found that for CelebA-HQ-220 HQQ2266
Figure 10: Effect temperature sampling in DDPM latents DiffuseVAF. (Top Row) Samples generated with x 1.0. (Bottom Row) Samples generated with X 0.8
G.6 DiffuseVAE Training Dynamics and Stability
Although hierarchical VAFs (Vahdat Kautz 2021: Child 2021 generate significantly better samples than standard VAE (with single stochastic layer) (Kingma & Welling 2014), the former can be unstable to train and often require carefully designed heuristics like spectral normalization, gradient clipping etc. However, even with these heuristics, stable training is not guaranteed. In contrast, standard VAEs often do not suffer from training instability issues. Indoed our empirical results in Figure 1I suggest the same. Figure I1 shows VAE training dynamics during training for the CIFAR-10 (Top Row) and the CelebA-HQ 256 (Bottom Row) datasets. As expected, the reconstruction loss (Middle column) and the total loss (Right column) for both the datascts docrease as training progresses. On the other hand, the KL loss increases for both the datascts early during training (Left column). This can be expected since during training, the VAE posterior q(2(1:) becomes complex to obtain better reconstruction loss. Therefore, the divergence
30

---

Published in Transactions on Machine Learning Research (11/2022)
KL LoG
190me
Reccon Lose
Total Losa
ColetsAHQ-756
13644
60-02
2008
Figure 11: Illustration of VAE training dynamics on the CIFAR-10 (Top Row) and the CelebA-HQ 256 dataset (Bottom Row) datascts. The columns from left to right represent the variation in KL loss, Recon- struction Loss and Total Loss during training respectively.
between (<=(x) and the prior p(z) (in our case a standard gaussian) increases, leading to a higher KT. Loss. Therefore, DiffuseVAE more stable to train than the corresponding hierarchical VAE and GAN-based counterparts.
G.7 Learning curve comparison between DiffuseVAE formulations
Figure shows comparison between the learning curves between DiffuseVAE formulations for the CIFAR-10 and CelebA-HQ 128 benchmarks. We used this analysis assess model convergence. For the CTFAR-10 dataset, our models started to slightly overfit after 2000 epochs, so we utilize the correspouding checkpoint for all analysis. For the CelebA-HQ 128 dataset, stopped training after exhaustion of our maximun compute budget of 1000 epochs and utilize the corresponding checkpoint for subsequent analysis and comparisons.
CIFAR-10
CelebAHQ-128
26-2
FID0910X
FID@1kk
1.04
243
Epochs
Form-1
Form22
Figure 12: Learning curve (FID vs epochs) comparison between DiffuseVAA, formulations for the CIFAR-10 (Left) and the CelebA-HQ-128 dataset (Right). T-1000 during inference
G.8 Additional Samples
To demonstrate generalization to more complex scenes, qualitative samples generated using Diffu- seVAE model trained on the LHQ-256 datasct with T=1000 and a temperature scaling factor of 0.8 are shown in Figure 17 Some additional samples from our CelebA-HQ model using DDIM sampling with 50 steps in the reverse process are shown in Figure [8] Figure 19 shows some additional samples from the same model with T=1000 and a temperature scaling factor of 0.8. Lastly, Figure 20 shows samples from the CelebAHQ-22 model but with shared latents in the DDPM stage (so effectively the generation is driven completely by low-dimensional latents from the VAE model).
31

---

Published in Transactions on Machine Learning Research (11/2022)
CIFAR-10 (VAE)
CIFAR-10 (DiffuseVAE)
CelebaA-64 (VAE)
CelebA-64 (DiffuseVAE)
MM
CelebaA BieebaA-HQ-128 (VAE)
CelebaA DelebaA-HQ-128 (DiffuseVAE)
CelebaA-HQ-251 (VAE)
CelebaA-HO-225 (DiffuseVAE)
Figure 13: Additional results demonstrating 3ße generator-refiner framework in DiffuseVAE

---

Published in Transactions on Machine Learning Research (11/2022)
Original
Black Hair
Male
Pale Skin
Smiling
Young
Composite Edits
LI
Original
Smiling
Black Hair
Male
Eyeglasses
Goatee
Figure 14:
Additional results demonstrating controllable synthesis in the CelebA-HQQ-22 dataset. Green
boxes denote the vector addition operation while Red boxes denote the vector subtract operation
33

---

Published in Transactions on Machine Learning Research (11/2022)
VAE
DiffuseAEE
Receding
Original
Black Hair
Male
Young
Eyeglasses
Hairline
VAE
DiffuseVAE
Original
Smiling
Black Hair
Male
Eyeglasses
Goatee
Figure 15: Comparison between composite edit samples generated using the first stage VAE the corre- sponding refined samples generated by DiffuseVAE.
34

---

Published in Transactions on Machine Learning Research (11/2022)
8 x 8
16 16
32 32
a= 0.1 = 0.3
0.5
Figure 16: Illustration of DiffuseVAE generalization to different noise types in the conditioning signal CelebA-HQ-128. a denotes the standard deviation of the gaussian noise added to the conditioning signal. As noise becomes more severe, the output generated by DiffuseVAA. becomes significantly worse. All final samples generated using T=100 with DDIM sampling.
35

---

Published in Transactions on Machine Learning Research (11/2022)
Figure 17: Samples generated from DiffuseVAR, trained on the LHQ-256 dataset. 1=1000 during sampling
36

---

Published in Transactions on Machine Learning Research (11/2022)
a IQ
Figure 18: Additional samples from generated from DiffuseVAB trained on the CelebAHQ-25 dataset. 5-
using DDIM sampling.
37

---

Published in Transactions on Machine Learning Research (11/2022)
1
p
S
LU
Figure 19: Additional samples from generated from DiffuseVAF, trained on the CelebAHQ-256 dataset (T=1000, Temp. Scaling factor was set to 0.8 during sampling).
38

---

Published in Transactions on Machine Learning Research (11/2022)
E
Figure 20: Additional samples from generated from DiffuseVAF trained on the CelebAHQ-256 dataset with the DDPM latents shared between samples. The generation is effectively driven by low-dimensional VAE latent space (T=1000, Temp. Scaling factor was set to 0.8 during sampling). 39


Reviews:
[{"review_id": "r3P00fUR5s", "reviewer": "Reviewer_TFVU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n- The paper is well written. The main idea is clearly presented, in terms of both method description and experiment comparison.\n- To the best of my knowledge, DiffuseVAE is the first model to combine the two techniques: VAE and Diffusion models. Though the proposed idea is straightforward, but it comes a good motivation: Better speed vs quality tradeoff by combining the strengths of both models. This could be a useful data point to enrich the literature. \n- The experiments are conducted to support the above strengths to combine the two models: High image generation quality, faster diffusion process compared with baseline DDPM, and controllable generation with a latent code.\n\n\n**Weaknesses**\n- Novel insight is lacking. I am afraid that the readers who are familiar with the literature cannot learn much new after reading the paper. \n- Experiments are not exciting, though standard. The experiments are conducted are established datasets in the academy setting. The results are compared with some related models in terms of quantitative metrics such as FID & IS. The performance of DiffuseVAE is comparable to SoTA. \n- Lack of comparisons & discussion with existing papers. There are two types of works can be discussed and compared in the paper:\n\n  1. **VQVAE + Diffusion**. This paper combines standard VAE with diffusion models. It is straightforward to combine VQ-VAE with diffusion models, and perform the so-called two-stage training. In the literature, I'm afraid that similar ideas have been explored. For example, VQ-VAE is used to get the compact discrete representation of images. The diffusion process is applied in either the latent space or the pixel space. I strongly feel the proposed DiffuseVAE is closely related to this line of research, and should be thoroughly discussed and compared. Importantly,  VQVAE + Diffusion have been conducted in a larger scale and show more impressive empirical results and applications.\n  2. **More papers that report good results on these standard datasets**, eg CIFAR-10, CelebA-64. There are several generative models that achieve better numbers, especially GANs [1,2]. For example, In Table 11(b) of [1], the proposed ADA techqniues yields FID 2.42 and IS 10.14. Please consider include all methods and numbers in Table 11(b) of [1] in Table 4 of this paper. The current Table 4 is misleading in this sense:  GANs are not as bad as it is currently presented. \n\n\n[1] Training Generative Adversarial Networks with Limited Data, NeurIPS 2020\n\n[2] Feature Quantization Improves GAN Training, ICML 2020", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes DiffuseVAE, a model that combines VAE and diffusion models.  DiffuseVAE is trained in a two-stage manner: \n- Stage1: VAE Training: The VAE encoder takes the original image $x_0$ as input and generates a reconstruction $\\hat{x}_0$ \n- Stage 2: Denoising Diffusion Probabilistic Models (DDPM) Training: $\\hat{x}_0$ is used to condition the second stage DDPM.\n\nExperiments are conducted on standard image synthesis benchmarks like CIFAR-10,  CelebA-64 and CelebA-HQ.  It shows that DiffuseVAE exhibits synthesis quality comparable to recent state-of-the-art (SoTA) while maintaining access to a low-dimensional latent code representation.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Minimum adjustment:\n\n- Please revise the paper based on the weakness ``Lack of comparisons & discussion with existing papers''.\n\nGood-to-have adjustment:\n\n- Besides the obvious better trade-off as presented at its form, is there any other novel insight the authors can deliver? For example, (1) the impact of the latent space for diffusion process; (2) the diffusion process for better learning of a meaningful VAE latent space, any chance to improve the latent space collapse issue [1,2]? Does learned representation yield better discriminative power?\n\n[1] Avoiding Latent Variable Collapse with Generative Skip Models\n\n[2] Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing", "broader_impact_concerns": "The current description of *Broader Impact Statement* is good.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cdaxb2ovNW", "reviewer": "Reviewer_4eBL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The generator-refiner framework that uses a conditional diffusion model to refine the generated samples from VAE is an interesting idea. In particular, it combines the respective advantages of diffusion models and VAEs: diffusion models can generate high-quality images but they do not have a compact vector representations of images, while VAEs suffer from poor generation but they always have a fixed-length latent variable inferred from images for representation learning and attribute-based controllable generation. \n- For the diffusion model training, two formulations of conditional DDPM are proposed. The idea of Formulation II that conditions both the forward and backward processes on the VAE output seems novel and very interesting.\n- Experiments also showed that the generator-refiner framework can achieve better tradeoff between sample quality and number of steps. In particular, DiffuseVAE (with DDIM sampling) largely outperforms the standard unconditional DDIM for all $T \\leq 100$.\n- This work identifies the potential issue derived from the VAE training (i.e., the prior-hole problem) which may deteriorate the generation quality of DiffuseVAE, and proposes the post-fitting (with a GMM on the VAE latent space) method to improve the DiffuseVAE performance.\n\nWeaknesses:\n- This work starts from a more generalized case where the diffusion model conditions on a conditional signal $y$ and a latent representation $z$, but actually only considers the simplified setting for DiffuseVAE where $y=x_0$ and $z=\\hat{x}_0$. Why not directly start from the simplified case, which is how DiffuseVAE is modeled and evaluated? I think it will make your presentation more focused and consistent. \n- Although Formulation II for conditional DDPM is novel and interesting, its performance is not better than the previously used Formulation I across multiple experimental settings. This makes me wonder why we need Formulation II, which seems to be more complicated in training and sampling.  \n- For experiments on “state-of-the-art comparisons”, my major concern is whether DiffuseVAE can work well on more diverse, higher resolution datasets, such as ImageNet (res=64, 128 and 256). It would be more convincing to show comparable results with the latest state-of-the-art on ImageNet.\n- For experiments on “controllable generation”, the results are also not quite good. In Figure 6, we can see that many image edits are not quite disentangled across different features. For example, the “black hair” editing is not sufficient, and the “male” editing is entangled with “mouth opening”, etc. Also, in Figure 6, it would be good to add the editing results of training VAE only to see where the potential failures/sufferings in image editing come from.  \n- For the experiments on generalizing to different noise types, I think the results (only on CIFAR-10 with some qualitative visualizations) are not quite convincing. First, in what conditions does your method perform well? For example, how large does the noise strength would be? What is the range for blurriness severity? Second, do we have similar results on CelebA-HQ? \n\nMinor concerns:\n- When “handling the DDPM stochasticity” by “sharing all stochasticity in the DDPM reverse process across all generated samples”, I wonder how to do it? Do you convert to a ODE-like deterministic sampling?\n- In Table 4, what does “LSGM (FID)” mean? I didn’t see an explanation of how it differs from the standard LSGM.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposed a generator-refiner framework, termed DiffuseVAE, that unifies the VAE and diffusion model for high-quality image generation. The DiffuseVAE training consists of two stages: 1) the standard VAE training that takes the original image as input and generates a reconstructed image, and 2) the diffusion model (i.e., DDPM) training by conditioning on the reconstructed images from VAE. Specifically, two formulations for the conditional diffusion model are proposed, where the first formulation only conditions the reverse denoising process on the VAE reconstruction (which has been commonly used in previous works), and the second formulation conditions both the forward diffusion and reverse denoising processes on the VAE reconstruction. Through extensive experiments on CIFAR-10, CelebA-64, and CelebA-HQ, this work demonstrates the effectiveness of the proposed method in generating high-quality samples, equipping DDPM with a low dimensional latent code  for controllable generation, and providing a better tradeoff between sample quality and number of steps. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see the weaknesses (which are critical to securing my recommendation) and minor concerns in the above.", "broader_impact_concerns": "I think the potential applications and the negative impacts have been well discussed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8ve0qvZ4L5", "reviewer": "Reviewer_5cpv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper can be summarized as follows:\n\n- The manuscript is well-written except a few minor problems in the statements. A clear review of the diffusion models was also given in the supplementary material.\n\n- In the theoretical analyses, derivations of objections were given in detail.\n\n- Some of the implementation details, such as network architectures and hyperparameters, were given.\n\n- In the analyses, the DiffuseVAE outperforms VAE based methods.\n\nOne of the main weaknesses of the paper is the lower performance of DiffuseVAE compared to the state-of-the-art. Although DiffuseVAE boosts accuracy of VAEs and lighter than state-of-the-art higher performing models, its accuracy may need to be further improved to compete with the state-of-the-art.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents DiffuseVAE, which integrates VAE within a diffusion model, to design novel conditional parameterizations for diffusion models. In the experimental analyses, the proposed DiffuseVAE was examined in downstream tasks like controllable synthesis. The proposed method  improves the speed-quality tradeoff exhibited in standard unconditional DDPM/DDIM models. On  image synthesis benchmarks, the proposed model outperforms most existing VAE-based methods while performing on par with state-of-the-art models.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Training VAEs are challenging in general. Could you please elaborate training difficulties/challenges of the proposed DiffuseVAE and provide details regarding the employed tricks?\n\n- Could you please provide learning curves for some of the analyses?\n\n- DiffuseVAE is lighter than state-of-the-art by model size. Could you please also compare running times for inference and training of models?\n\n- Qualitative results were given mainly for face images and low-resolution Cifar-10 images. Could you please provide additional results for higher resolution images containing other objects, such as for some samples from the Imagenet (even 128x128 are fine), scenes, etc. to evaluate generalization of the DiffuseVAE for larger class of objects.", "broader_impact_concerns": "A Broader Impact Statement was given and some concerns were discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed a new combination of a VAE and a Diffusion-based model. It is the first (or at least one of the first) approaches to combine these two techniques. The paper is well-written and easy to follow. The results are mostly convincing and support the initial claims. After discussing the decision with the reviewers, I recommend to accept the paper as it is.
Decision: Accept (poster)