Score-based Generative Modeling in Latent Space
Authors: Arash Vahdat
Karsten Kreis
Jan Kautz
Venue: NeurIPS 2021 Poster (2021)
Keywords: score-based generative models
variational autoencoders
image synthesis
deep generative models

Abstract: Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. 


Score-based Generative Modeling in Latent Space
Arash Vahdat" NVIDIA
Karsten Kreis"
Jan Kautz NVIDIA
NVIDIA
avahdat@nvidia.com
kkreistnvidia.com
jkautzónvidia.com
Abstract
Score-based generative models (SGMs) have recently demonstrated impressive results terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Model (LSGM), a novel approach that trains SGMs in latent space, relying on the variational autoencoder framework. Moving from data latent space allows us train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in smaller space, resulting in fewer network evaluations and faster sampling. To enable training ISGMMs end-to-end in scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterizztion of the score function that allows SGM to focus on the mismatch of the target distribution with respect to simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of oef-the-art FID score of 2.10 on CIFAR- 10, outperforming all existing generative results this dataset. On CelebA- -alb--------0 HQQ256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our implementation is available at https: htap:::///tttin //github....... www.coom/aalaeeeecccocoooooooeae /WW//vallssssaaeeee
Generative
Introduction
The long-standing goal of likelihood-based generative learning is to faithfully learn a data distribution, while also generating high-quality samples. Achieving these two goals simultaneously is tremendous challenge, which has led to the development of a plethora of different generative models. Recently, score-based generative models (SGMs) demonstrated astonishing results terms of both high sample quality and likelihood [1, 2]. These models define a forward diffusion process that maps data to noise by gradually perturbing the imput data. Generation corresponds to a reverse process that synthesizes novel data via iterative denoising, starting from random noise. The problem then reduces to learning the score function the gradient of the log-density of the perturbed data [3]. seminal work, Song et al. [2] show how this modeling approach is described with stochastic differential equation (SDE) framework which can be converted to maximum likelihood training [4]. Variants of SGMs have been applied images [1, 2, 5, 6), audio [7, 8, 9, 10), graphs [11] and point clouds [12, 13].
Albeit high quality, sampling from SGMs is computationally expensive. This is because generation amounts to solving complex SDE, or equivalently ordinary differential equation (ODE) (denoted as the probability flow ODE in [2]), that maps a simple base distribution to the complex data distribution. The resulting differential equations
typically complex and solving them accurately requires numerical integration with very small step sizes, which results in thousands of neural network evaluations [1, 2. 6]. Furthermore, generation complexity is uniquely defined by the underlying data distribution and the forward SDE for data perturbation, implying that synthesis speed cannot be
"quual contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Encoder
p(zo)
Latent Space Diffusion
p &1
Datax
g /x)
SDE Trajectories
Probubility Flow
Reconst, p(x(zo)
Decoder
(L(( (
Latent Space Denoising
Figure 1: our latent score-based generative model (LSGM), data is mapped to latent space via an encoder g(zo|x) and a diffusion process is applied in the latent space (Zu + 2.). Synthesis starts from the base distribution p(z;) and generates samples in latent space via denoising (Zo + Z1). Then, the samples are mapped from latent to data space using decoder p(x|zo). The model is trained end-to-end.
increased easily without sacrifices. Moreover, SDE-based generative models are currently defined for continuous data and cannot be applied effortlessly to binary, categorical, or graph-structured data.
Here, we propose the Latent Score-based Generative Model (LSGM), new approach for learning SGMs in latent space, leveraging a variational autoencoder (VAE) framework [14, 15]. We map the input data to latent space and apply the score-based generative model there. The score-based model is then tasked with modeling the distribution over the embeddings of the data set. Novel data synthesis is achieved by first generating embeddings via drawing from a simple base distribution followed by iterative denoising, and then transforming this embedding via a decoder to data space (see Fig. 1). We can consider this model VAE with an SGM prior. Our approach has several key advantages:
Synthesis Speed: By pretraining the VAE with a Normal prior first, we can bring the marginal distribution over encodings (the aggregate posterior) close to the Normal prior, which is also the SGM's base distribution. Consequently, the SGM only needs to model the remaining mismatch, resulting in less complex model from which sampling becomes easier. Furthermore, we tailor the latent space according to our needs. For example, we can use hicrarchical latent variables and apply the diffusion model only over a subset of them, further improving synthesis speed.
Training a regular SGM can considered as training a neural ODE directly on the data [2]. However, previous works found that augmenting neural ODEs [16, 17] and more generally generative models [18, 19, 20, 21] with latent variables improves their expressivity. Consequently, expect similar performance gains from combining SGMs with latent variable framework.
Expressivity:
Tailored Encoders and Decoders: Since we use the SGM in latent space, we can utilize carefully de- signed encoders and decoders mapping between latent and data space, further improving expressivity. Additionally, the L.GMM method can therefore be naturally applied to non-continuous data.
LSGMs can be trained end-to-end by maximizing the variational lower bound on the data likelihood. Compared to regular score matching, our approach comes with additional challenges, since both the score-based denoising model and its target distribution, formed by the latent space encodings. are learnt simultaneously. To this end, we make the following technical contributions: (i) We derive a new denoising score matching objective that allows us to efficiently learn the VAE model and the latent SGM prior the same time. (ii) We introduce a new parameterization of the latent space score function, which mixes a Normal distribution with learnable SGM, allowing the SGM to model only the mismatch between the distribution of latent variables and the Normal prior. (iii) We propose techniques for variance reduction of the training objective by designing new SDE and by analytically deriving importance sampling schemes, allowing us to stably train deep LSGMs. Experimentally, we achieve state-of-the-art 2.10 FID on CIFAR-10 and 7.22 FID CelebA-HQ-255.. and significantly improve upon likelihoods of previous SGMs. On CelebA-HQ-24 previous SGMs in synthesis speed by two orders of magnitude. We also model binarized images, MNIST and OMNIGLOT, achieving offt-oo--he-ar likelihood on the latter.
we outperform
Background
Here, we review continuous-ting acore-based generative models (see [2] for an in-depth discussion). Consider a forward diffusion process [2, |== for continuous time variable E [0. 1). where Zn is the starting variable and 2, its perturbation time f. The diffusion process is defined by an Itô SDE:
dz = ft)z df + g(t) dw
(1)
where f:R. + and g - IR are scalar drift and diffusion coefficients, respectively, and w is the standard Wiener process. f(t and g(t) can be designed such that z NN(2;; 1, follows

---

Normal distribution at the end of the diffusion process." 2 Song et al. [2] show that the SDE in Eq. 1 can be converted to generative model by first sampling from 2 N (z) 0. I) and then running the reverse-time SDE dz = [f(1)z=g1 g((()Vx
x log gi (z) dt + (()) dw, where w is a reverse-time standard Wiener process and dł is an infinitesimal negative time step. The reverse SDE requires knowledge of
V 2; log (z,), the score function of the marginal distribution under the forward diffusion at time t.
One approach for estimating it is via the score matching objective*:
min [0 ((o) (mo) Vll log ((z)) Vz, log
(2)
that trains the parameteric score function Vz. log pe (Zz) at time ~ 24(0, 1] for a given weighting coefficient X(t). (((o) is the zo tu-generating distribution and a(z Zo) is the diffusion kernel, which is
available in closed form for certain ft) and g().. Since Vz, log ((,) is not analytically available, Song et al. [2] rely on denoising score matching [22] that converts the objective in Eq. 2 to:
min (. ((t)B Blyze (z.) V2, log ((2z) Vz, log Po (z+ (z) 0
+ +
(3)
Vincent [22] shows C = E1
1ooe:oo,_((()AAA (A(1) Egte, | Vz, log ((x))()))))) 313 ||V,. n. log g [1x]] is
independent of 9, making the minimizations in Eq. 3 and Eq. 2 equivalent. Song et al. [4] show that for A(t) = g(t)2/2, the minimizations correspond to approximate maximum likelihood training (KI.) divergence between the target distribution and the distribution defined by the reverse-time generative SDE with the learnt score function. In particular, the objective of Eq. 2 can then be written:
based on an upper on the Kullback-Leibe
g(t)2
KL (g(2)))))l (2o)) & 10
2 Egtno: Egtre [IV a. log g(ze) Vz, log po (z:) (2)]
(4)
which can again be transformed into denoising score matching (Eq. 3) following Vincent [22].
3 Score-based Generative Modeling in Latent Space
The LSGM framework in Fig. 1 consists of the encoder (z( xx), SGM prior po (zo). and decoder Pv (x/2o). The SGM prior leverages a diffusion process as defined in Eq. and diffuses Zu ~ q Zu xx) samples in latent space to the standard Normal distribution p(z,) ) = =N(z;; 0, Generation uses the reverse SDE to sample from po (o())) with time-dependent score function Vz, log and the
decoder Pv (x (20) to map the synthesized encodings 2n data space. Formally, the generative process
is written as p( x) = po po(zo) P (x 22)) The goal of training is to learn (Φ) A 1)). the parameters of the encoder go (zo [x). score function Vz, log pe (Zz), and decoder pv (x|zo). respectively.
We train LSGM by minimizing the variational upper bound on negative data log-likelihood log p(x):
C(x, db. 0. b)
Bag (2))xx [- Po (x ]
KL (20 pa (zu))
(S)
E (x0)x) |- log (x(z))
+ (mo1x) [log go + +B (malxx) |- log po (z)))
(6)
rocomeruction scrm
zegative eacoder entropy
cross entropy
following a VAE approach [14, 15], where q (Zolx) approximates the true posterior p(zn) xx).
In this paper, we use Eq. 6 with decomposed KI. divergence into its entropy and cross entropy terms. The reconstruction and entropy terms are estimated easily for any explicit encoder as long as the reparameterization trick is available [14]. The challenging part in training LSGM is to train the cross entropy term that involves the SGM prior. We motivate and present our expression for the cross-entropy term in Sec. 3.1, the parameterization of the SGM prior in Sec. 3.2, different weighting mechanisms for the training objective in See. 3.3, and variance reduction techniques in Sec. 3.4.
3.1 The Cross Entropy Term
One may ask, why not train LSGM with Eq. 5 and rely on the KL in Eq. 4. Directly using the KL expression in Eq. 4 is not possible, as it involves the marginal score Vz, log (zz). which is unavailable analytically for common non-Normal distributions ((z)) such as Normalizing flows.
'tther distributions at = 1 are possible; for instance, see the "variance-exploding" SDE [2]. In this paper, however, we use only SDEs converging towards A (21; 0, I) at = 1.
We omit the t-subscript of the diffuused distributions gr in all score functions of the form Vz, log (z)z)
3

---

Transforming into denoising score matching does not help either, since in that case the problematic Zz log (Zz)) term appears the term (see Eq. 3). contrast to previous works [2, 22]. we cannot simply drop C, since it is, in fact, not constant but depends on g(Z+), which is trainable in our setup.
To cireumvent this problem, we instead decompose the KL in Eq. 5 and rather work directly with the cross entropy between the encoder distribution g(zo|x) and the SGM prior p(zo). We show:
Theorem 1. Given two distributions q( (zoox) and p(zo), defined in the continuous space RD, denote the marginal distributions of diffused samples under the SDE in Eq. 1 at time t with q(z,|x) and p(zi). Assuming mild smoothness conditions on log ((z)(x)) and log p(zz), the cross entropy is:
CE(g(zolx)) ((())) = E2-
9(1)² 2
20)))))))))
[[IV log q(z)/2)) - Ue, logp(z))18)
+ -
D log (2ne0d)
with g(ze, zolx) - g(zz) (z) (g x) and a Normal transition kernel q(z,) (0) = N (N u4 (20) II where be and of ure obtained from ((1) and g(t) for fixed initial variance ob at = O.
A proof with generic expressions for b4 and of well an intuitive interpretation are in App. A.
Importantly, unlike for the KL objective of Eq. 4, no problematic terms depending on the marginal score Vz. x, log (z, x) arise. This allows us to use this denoising score matching objective for the cross entropy term in Theorem not only for optimizing p(zo) (which is commonly done in the score matching literature), but also for the g(2o)* x) encoding distribution, It can be used even with complex (( x) distributions, defined, for example, in a hierarchical fashion [20, 21] or via Normalizing flows [23, 24]. Our novel analysis shows that, for diffusion SDEs following Eq. 1, only the cross entropy can be expressed purely with V2, log q Zr (z)). Neither KL nor entropy in [4] can be expressed without the problematic term Vz, log (Z) x) (details in the Appendix).
Note that in Theorem I, the term Vz, log p(z)) in the score matching expression corresponds to the score that originates from diffusing an initial p(zo) distribution. In practice, we use the expression to learn an SGM prior Po Po(Zo) (z)), which models V.. log p(z.) by a neural network. With the learnt score Vz, log po (2x) (here we explicitly indicate the parameters 0 to clarify that this is the learnt model), the actual SGM prior is defined via the generative reverse-time SDE (or, alternatively, a closely-connecter ODE, see Sec. 2 and App. D), which generally defines its own, separate marginal distribution po (2o) at t = 0. Importantly, the learnt, approximate score Vz, log pe po (zr) is not necessarily the same as one would obtain when diffusing po po (Zo). Hence, when considering the learnt score Vz, log po(z,) the score matching expression in our Theorem only corresponds to an upper bound on the cross entropy between (zu xx) and pe ( defined by the generative reverse-ting SDE. This is discussed in detail in concurrent works [4, 25]. Hence, from the perspective of the learnt SGM prior, we are training with an upper bound on the cross entropy (similar to the bound on the KL in Eq. 4), which can also be upper
considered as the continuous version of the discretized variational objective derived by Ho et al. [1].
3.2 Mixing Normal and Neural Score Functions
In VAEs [14], p(zo) is often chosen as a standard Normal (azo I). For recent hierarchical VAEs [20, 21], using the reparameterization trick, the prior can be converted to N(zo 0, I) (App. E).
Considering a single dimensional latent space, we can assume that the prior at time i is in the form of geometric mixture p(=;) (z,;0, 1))=-,,()))) where (2x) is a trainable SGM prior and a E [0. 1] is learnable scalar mixing coefficient. Formulating the prior this way has crucial advantages: (i) We can pretrain LSGM's 's autoencoder networks assuming a=0, which corresponds to training the VAE with a standard Normal prior. This pretraining step will bring the distribution of latent variable close to A(2;; 0, 1), allowing the SGM prior to learn much simpler distribution in the following end-to-end training stage. (ii) The score function for this mixture is of the form Vz, log ((2)) = - (1 a)² + aV 2, log 1ogr'((e (z.). When the score function dominated by the linear term, we expect that the reverse SDE can be solved faster, as its drift is dominated by this linear term.
For our multivariate latent space, we obtain diffused samples at time t by sampling 2, ~ q(z, (zo) with z; = (4(2)) + o26, where 6 ~ NN(E;; O, I). Since we have Vz, log q( (zo) = -e/o,, similar to [1], we parameterize the score function by Vz, log pogz)))) == -ce(Z, t)/ where ca(z,, t) == o,(1 a) @ z, + a @ (z, t) is defined by our mixed score parameterization that is applied elementwise to the components of the score. With this, we simplify the cross entropy expression to:
CE(go(zu) xx) (zo))) B
[w(t))
(mlll) [[le ()) + log (2neot), (7)

---

where w(t) = /o? is time-dependent weighting scalar.
3.3
Training with Different Weighting Mechanisms
Table 1: Weighting mechanisms
The weighting term w(() in Eq. 7 trains the prior with maximum likelihood. Similar to [1, 2]. we observe that when wi(() is dropped while training the SGM prior (i.e., wn(t) = 1). LSGM often yields higher quality samples at small cost in likelihood. However, in our case, we can only drop the weighting when training the prior. When updating the encoder parameters, we still need to use the maximum likelihood weighting to ensure that the encoder ( x) is brought closer to the true posterior p(zolx)). Tab. summarizes three weighting mechanisms we consider in this paper: (n(() corresponds to maximum likelihood, Wun (t) the unweighted objective used by [1. 21, and Whe (t) is a variant obtained by dropping only 1/0 This weighting mechanism has a similar affect on the sample quality as Wan (1) = 1; however, in Sec. 3.4, we show that it is easier to define a variance reduction scheme for this weighting mechanism.
Mechanism Weights
Weighted (n((t) = g(t)) Unweighted Win =
Reweighted (i)(() s(1)²
The following summarizes our training objectives (with ~ ~2 1] and e ~ NN{; O,
min Exp(zalx) log Pio (x [20]] + (xalx)) [log go (20))x))) [x]] + (xx.e.(xill0)).)0 (wn(() 0.
||e ca co
(8)
2
Wh Wa/on/tte (t)
min (x.c
|| Eo (zi, 1 with ((z:/2o) = NNzs He (20). of II)
(9)
2
where Eq. trains the VAE encoder and decoder parameters {Φ, } using the variational bound [(x, σ, 0 4)) from Eq. 6. Eq. 9 trains the prior with one of the three weighting mechanisms. Since the SGM prior participates in the objective only in the cross entropy term, we only consider this term when training the prior. Efficient algorithms for training with the objectives are presented in App. G.
3.4 Variance Reduction
The objectives in Eqs. 8 and 9 involve sampling of the time variable t, which has high variance [26]. We introduce several techniques for reducing this variance for all three objective weightings. We focus on the "variance preserving" SDEs (VPSDEs) [2, 1, 27], defined by dz = ( dl + /B(I)dw dw where B(1)= = 3o + (81 - B linearly interpolates in [80, 31] (other SDEs discussed App. B).
We denote the marginal distribution of latent variables by g(zo) := (x))) ( Here, we derive variance reduction techniques for CE(g(zo) (p)()))))))))......))) assuming that both ((z)) = p(zo) = N(2o; O, I). This is reasonable simplification for our analysis because pretraining our LSGM model with a N (Z2; O, I) prior brings ((2)) close to (zo;; 0, I) and our SGM prior is often dominated by the fixed Normal mixture component. We empirically observe that the variance reduction techniques developed with this assumption still work well when (Zo) and p(zo) are not exactly N' (Z2 0, I).
Variance reduction for likelihood weighting: In App. B, for g = p(zo) N(zo: 0, 0). show (E(g(z))ll((z)) is given by Le log of /dt] + const. We consider two approaches: (1) Geometric VPSDE: To reduce the variance sampling uniformly from t, we can design the SDE such that d loo of /dt is constant for t E [0, 1). We show in App. B that a B(t) log(o /oa of (1-a?) with geometric variance of = ( (o / satisfies this condition. We call VPSDE with this 3(t) geometric VPSDE. 22 and o are the hyperparameters of the SDE, with 0 min <o <1 Although our geometric VPSDE has a geometric variance progression similar to the "variance exploding" SDE (VESDE) [2], it still enjoys the "variance preserving" property of the VPSDE. In App. B, we show that the VESDE does not come with reduced variance for t-sampling by default.
(2) Importance sampling (IS): We can keep B(t) and of unchanged for the original linear VPSDE, and instead use IS to minimize variance. The theory of IS shows that the proposal r(t) xx d log of/dl /dt has minimum variance [28]. In App. B, we show that we can sample from r(t) using inverse transform sampling = var ((of))
( (a)) (a 1-9) where var- is the inverse of of and ~ (1,0, 1]. This variance reduction technique is available for any VPSDE with arbitrary ((t).
In Fig. 2, we train a small LSGM on CIFAR-IO 10 with wn weighting using (i) the original VPSDE with uniform sampling, (ii) the same SDE but with our IS from f, and (iii) the proposed geometric
Minimizing C(x, do 0, 4b) w.n. & is equivalent to minimizing KL [((o w.n. g(oll))
5

---

VPSDE. Note how both (ii) and (iii) significantly reduce the variance and allow us to monitor the progress of the training objective. In this case. (i) has difficulty minimizing the objective due to the high variance. In App. B, we show how IS proposals can be formed for other SDEs, including the VESDE and Sub- VPDDE from [2].
Variance reduction for unweighted and reweighted objectives: When training with Won, analytically deriving IS proposal distri- butions for arbitrary B(t) is challenging. For linear VPSDEs, we provide a derivation in App. to obtain the optimal IS distribution. In contrast, defining IS proposal distributions is easier when training with wie. In App. B, we show that the optimal distribution is in the form r(t) x do, /df which is sampled by t= varr ((1- + p with p~ 4 1]. In Fig. 3, we visualize the IS distributions for the three weighting mechanisms for the linear VPSDE with the original [8a, [1] parameters from [2]. r(t) for the likelihood weighting is
5030
VPSDE, Lin. VPSDE, IS VPSDE, Un
Ttuining N81.30
0G30
7930
7030
100
22.0
400
Epochs
Figure 2: Variance reduction
weighted
more tilted towards t = 0 due to the 1/ož term in wn.
unweighted
reweigtted
When using differently weighted objectives for training, we can either sample separate with different IS distributions for each objective, or use IS for the SGM objective (Eq. 9) and reweight the samples according to the likelihood objective for encoder training (Eq. 8). See App. G for details.
0.88
Figure 3: IS distributions
4 Related Work
Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 371, specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion- based objectives [3, 38, 1, 2. 6]. Among those, [1, 6] use a discretized diffusion process with many noise scales, building on [27], while Song et al. [2] introduce the continuous time framework using SDEs. Experimentally. these works focus on image modeling and, contrary to us, work directly in pixel space. Various works recently tried to address the slow sampling of these types of models and further improve output quality. [39] add an adversarial objective, [5] introduce quality, and sample diversity, [40] learn a sequence of conditional energy- based models for denoising, [41] distill the iterative sampling process into single shot synthesis, and [42] learn an adaptive noise schedule, which is adjusted during synthesis to accelerate sampling. Further, [26] propose empirical variance reduction techniques for discretized diffusions and introduce new, heuristically motivated, noise schedule. In contrast, our proposed noise schedule and our variance reduction techniques are analytically derived and directly tailored our learning setting in the continuous time setup.
Markovann diffusion processes that allow to trade off synthesis speed,
Recently, [11] [1]] presented a method to generate graphs using score-based models, relaxing the entries of adjacency matrices to continuous values. LSGM would allow to model graph data more naturally using encoders and decoders tailored to graphs [43, 44, 45, 46).
Since our model can be considered VAE [14, 15] with score-based prior, it is related to approaches that improve VAE priors. For example, Normalizing flows and hierarchical distributions [23, 24, 47, 48, 20, 21], as well as enargy-based models [49, 50, 51, 52, 53] have been proposed as VAE priors. Furthermore, classifiers [54, 55, 56), adversarial methods [57], and other techniques [58, 59] have been used to define prior distributions implicitly. In two-stage training, separate generative model trained in latent space as a new prior after training the VAE itself [60, 61, 62, 63, 64, 10]. Our work also bears resemblance to recent methods on improving the sampling quality in generative adversarial networks using gradient flows in the latent space [65, 66, 67, 68]. with the main difference that these prior works use a discriminator to update the latent variables, whereas we train an SGM.
Concurrent works: [10] proposed learn denoising diffusion model in the latent space of a VAE for symbolic music generation. This work does not introduce an end-to-end training framework of the combined VAE and denoising diffusion model and instead trains them in two separate stages. In contrast, concurrently with us [69] proposed an end-to-end training approach, and [70] combines contrastive learning with diffusion models in the latent space of VAES for controllable generation. However, [10, 69, 70] consider the discretized diffusion objective [1]. while we build on the continu- ous time framework. Also, these models are not equipped with the mixed score parameterization and variance reduction techniques, which we found crucial for the successful training of SGM priors.

---

Additionally, [71, 4, 25] concurrently with us proposed likelihood- training of SGMs in data space'. [4] developed a bound for the data likelihood in their Theorem 3 of their second version, using a denoising score matching objective, closely related to our cross entropy expression. However, our cross entropy expression is much simpler as we show how several terms can be marginalized out analytically for the diffusion SDEs employed by us (see our proof in App. A). The same marginalization can be applied to Theorem 3 in [4] when the drift coefficient takes a special affine form (i.e., f(z, t) f(t)z). Moreover, [25] discusses the sxelihood-basee training of SGMs from a fundamental perspective and shows how several score matching objectives become a variational bound on the data likelihood. [71] introduced notion of signal-to-none to-nose ratio (SNR) that results in invarrinv parameterization of time that depends only on the initial and final noise. Interestingly, our importance sampling distribution in Sec. 3.4 has similar noise-invariant parameterization of time via /. = var ((o))" 1-), which also depends only on the initial and final diffusion process variances. We additionally show that this time parameterization results in the optimal minimum- variance objective, if the distribution of latent variables follows a standard Normal distribution. Finally, [72] proposed a modified time parameterization that allows modeling unbounded data scores.
5 Experiments
Here, we examine the efficacy of LSGM in learning generative models for images.
Implementation details: We implement LSGM using the NVAE [20] architecture as VAE backbone and NCSN++ [2] SGM backbone. NVAE has a hierarchical latent structure. The diffusion process input Zo is constructed by concatenating the latent variables from all groups in the channel dimension. For NVAEs with multiple spatial resolutions in latent groups, we only feed the smallest resolution groups to the SGM prior and assume that the remaining groups have standard Normal distribution.
Sampling: To generate samples from LSGM at test time, use black-box ODE solver [73] to sample from the prior. Prior samples are then passed to the decoder to generate samples in data space.
Evaluation: We measure NELBO, an upper bound on negative log- ligeliieldoood (NLL), using Eq. 6. For estimating log p(zo), we rely on the probability flow ODE [2], which provides an unbiased but stochastic estimation of log p(z)). This stochasticity prevents us from performing an importance weighted estimation of NLL [74] (see App. for details). For measuring sample quality, Fréchet inception distance (FID) [75] is evaluated with 50K samples. Implementation details in App. G.
5.1 Main Results
Unconditional color image generation: Here, we present our main results for unconditional image generation on CIFAR-10 [89] (Tab. 2) and CelebA-HQ 256 (5-bit quantized) [88] (Tab. 3). For CIFAR-10, we train 3 different models: LSGM (FID) and LSGM (balanced) both use the VPSDE with linear (() and -wi-whhghting for the SGM prior in Eq. 9, while performing IS as derived in Sec. 3.4. They only differ how the backbone VAE is trained. LSGM (NLL) is a model that is trained with our novel geometric VPSDE, using wi-weighting in the prior objective (further details in App. G). When set up for high image quality, LSGM achieves a new otate-of-the-art FID of 2.10. When tuned towards NLL, achieve NELBO of 2.87, which is significantly better than previous score-based models. Only autoregressive models, which come with very slow synthesis, and VDVAE [21] reach similar or higher likelihoods, but they usually have much poorer image quality.
For CelebA-HQ-2566 we observe that when LSGM is trained with different SDE types and weighting mechanisms, it often obtains similar NELBO potentially due to applying the SGM prior only to small latent variable groups and using Normal priors at the larger groups. With wre-weighting and linear VPSDE, LSGM obtains the state-of- the FID score of 7.22 on par with the original SGM [2).
For both datasets, we also report results for the VAE backbone used in our LSGM. Although this baseline achieves competitive NLL, its sample quality is behind our LSGM and the original SGM.
Modeling binarized images: Next, we examine LSGM on dynamically binarized MNIST [93] and OMNIGLOT [74]. We apply LSGM to binary images using a decoder with pixel-wise independent Bernoulli distributions. For these datasets, we report both NELBO and NLL in nats in Tab. 4 and Tab. 5. On OMNIGLOT, LSGM achieves state-of-the-a likelihood of $87.79 nat, outperforming previous models including VAßs with autoregressive decoders, and even when comparing its NELBO
We build on the V1 version of [4], which was substantially updated after the NeurIPS submission deadline.
7

---

Table 2: Generative performance CIFAR-IO.
Table 3: Generative results on CelebA-HQ-256.
Method
NLL
FID
Method
NLL.
FID
I.GMM (FID)
$3.43
2.10
LSGM
<0.70
7.22
Ours
Ours
LSGM (NLL)
<2.87
6.89
VAE Backbone
0.70
30.87
LSGM (balanced)
$2.95
2.17
NVAE
0.70
29.76
VAE Backbone
2.96
43.18
VAEBM [76]
20.38
VAES
VDVAE [21]
2.87
NCP-VAE [56]
24.79
NVAE [20]
2.91
23.49
DC-VAE [77]
15.80
VAEBM [76]
12.19
VAEs
Score
SDE [2]
7.23
NCP-VAE [56]
24.08
BIVA [48]
3.08
Flows
GLOW [85]
1.03
68.93
DC-VAE [77]
17.90
Aut. Reg.
SPN [86]
0.61
NCSN [3]
25.32
Rec. Likelihood [40]
3.18
9.36
Adv. LAE [87]
19.21
DSM-AL.S [39]
3.65
GANs
VQ-GAN [64]
10.70
Score
DDPM [1]
3.75
3.17
PGGAN [88]
8.03
Improved DDPM [26)
2.94
11,47
SDE (DDPM++) [2]
2.99
2.92
SDE (NCSN++)
2.20
12
VFlow [19]
2.98
60
Flows
ANF [18]
3.05
DistAug aug [78]
2.53
42.90
20
Sp. Transformers [79]
2.80
10-2
Aut. Reg. VAE [80]
2.83
ODE Solver Error Tolerance
PixeISNAIL [81]
2.85
Figure 4: FID and number of function evaluations (NFEs) for different ODE solver error tolerances CelehA-HQ-254. LSGM takes 4.15 see. for sampling while the original SGM [2) takes 45 min. with PC and 3.9 min. with ODF-based sampling.
PixeICNN++ [82]
2.92
AutoGAN [83]
12.42
GANs
StyleGAN2-ADA [84]
2.92
Table 4: Dyn. binarized OMNIGLOT results.
Table 5: Dynamically binarized MNIST results.
Method
NELBOJ
NLL,
Method
NELBOJ
NLL
Ours
LSGM
87.79
<87.79
Ours
LSGM
78.47
778.47
NVAE [20]
93.92
90.75
NVAE [20]
79.56
78.01
BIVA [48]
93.54
91.34
BIVA [48)
80.06
78.41
VAEs
VAEEs
DVAE+ [51]
92.38
IAF-VAE [24]
80.80
79.10
Ladder VAE [90]
102.11
DVAE+ [51]
78.49
VLVAE (47)
89.83
PixeIVAE++ [91]
78.00
Aut. Reg.
VampPrior [59]
89.76
Aut. Reg.
VampPrior [59]
78.
PixeIVAE++ [91]
88.29
MAE [92]
77.98
against importance weighted estimation of NLL for other methods. On MNIST, LSGM outperforms previous VAEs in NELBO, reaching NELBO 1.09 nat lower than the state-of-the-ard NVAE.
Qualitative results: We visualize qualitative results for all datasets Fig. On the complex multimodal CIFAR-10 dataset, LSGM generates sharp and high-quality images. On SelebA-HHQ2556 LSGM generates diverse samples from different ethnicity and age groups with varying head poses and facial expressions. On MNIST and OMNIGLOT, the generated characters are sharp and high-contrast.
Sampling time: We compare LSGM against the original SGM [2] trained on the CelebA-HQ-255 dataset in terms of sampling time and number of function evaluations (NFEs) of the ODE solver. Song et al. [2] propose two main sampling techniques including predictor-corrector (PC) and probability flow ODE PC sampling involves 4000 NFEs and takes 44.6 min. on Titan V for batch of 16 images. yields 7.23 FID score (see Tab. 3). ODE-based sampling from SGM takes 3.91 min. with 335 NFEs, but it obtains poor FID score of 128.13 13 with 10-5 ODE solver error tolerance'.
In stark contrast, ODE-based sampling from our LSGM takes 0.07 min. with average of 23 NFEs. yielding 7.22 FID score. LSGM is 637x x and 56x faster than original SGM's [2] PC and ODE
com/yang song/ sttrs/ccceeeeeeeeee sdde sd___dd_________ sicee_pytrcchh. Song et al. [2] report that ODE-based sampling yields worse FID scores for their models (see D.4 in (2]). The problem is more severe for VESDEs. Unfortunately, submission time only VESDE model released.
"We use the VESDE checkpoint at

---

7
6
10
3
57
(c) OMNIGLOT
/ O
7
4
8
2
0
7
2
5
0
5 7 8
6
2
3
0
7
3
6
7
5
0
5
5
4
9
(a) CIFAR-10
(b) CelebA-HQ-2566
MNIST
Figure 5: Generated samples for different datasets. For binary datasets, visualize the decoder mean. LSGM successfully generates sharp, high-quality, and diverse samples (additional samples appendix).
Table 6: Ablations SDEs, objectives, weighting mechanisms, and variance reduction. Details in App.
weiihhjj
Was
Whe
(sampling (SGM-obj.)
24 1
ni(()]
24 [O,
(t)
24 24 1]
(t)
1-sampling (q-obj.)
rew.
rew
rew.
rn())
rew.
ru())
new:
ru(e)
rew:
ru(t)
Geom.- FID
10.18
n/a
NaN
NaN
n/a
n/a
22.21
NaN NoN
7.29
7.18
VPSDE
NELBOJ
2.96
NaN
NaN
n/a
3.04
2.99
2.99
FIDJ
6.15
8.00
NaN
NaN
5.39
5.39
NaN
4.99
15.12
6.19
VPSDE
NELBOJ
2.97
2.97
NaN
NaN
2.98
2.98
NaN
2.99
3.03
2.99
sampling, respectively. In Fig. 4, we visualize FID scores and NFEs for different ODE solver error tolerances. Our LSGM achieves low FID scores for relatively large error tolerances.
We identify three main reasons for this significantly faster sampling from LSGM: (i) The SGM prior in our LSGM models latent variables with 32x32 spatial dim., whereas the original SGM [2] directly models 256 256 images. The larger spatial dimensions require deeper network to achieve large receptive field. (ii) Inspecting the SGM prior in our model suggests that the score function is heavily dominated by the linear term the end of training, the mixing coefficients a are all < 0.02. This makes our SGM prior smooth and numerically faster to solve. (iii) Since SGM is formed in the latent space in model, errors from solving the ODE can be corrected to some degree using the VAE decoder, while in the original SGM [2] errors directly translate to artifacts in pixel space.
5.2 Ablation Studies
SDEs, objective weighting mechanisms and variance reduction. In Tab. 6, we analyze the differ- ent weighting mechanisms and variance reduction techniques and compare the geometric VPSDE with the regular VPSDE with linear B(t) [1. 2]. In the table, KGM-objj-weightine ing mechanism used when training the SGM prior (via Eq. 9). t-sampling (SGM-obj..) indicates the sampling approach for f, where ru(t), Fun (t) and (t) denote the IS distributions for the weighted (likelihood), the unweighted, and the reweighted objective, respectively. For training the VAE encoder (zo))) (last term in Eq. 8), we either sample a separate batch with importance sampling following mi()) (only necessary when the SGM prior is not trained with un itself), or we reweight the samples drawn for training the prior according to the likelihood objective (denoted by rew.). n/a indicates fields that do not apply: The geometric VPSDE has optimal variance for the weighted (likelihood) objective already with uniform sampling; there is no additional IS distribution. Also, we did not derive IS distributions for the geometric VPSDE for Wan NaN indicates experiments that failed due to training instabilities. Previous work [20, 21] have reported instability in training large VAEs. We find that our method inherits similar instabilities from VAEs; however, importance sampling often stabilizes training our LSGM. As expected, we obtain the best NELBOs (red) when training with the weighted, maximum likelihood objective (wn)). Importantly, our new geometric VPSDE achieves the best NELBO. Furthermore, the best FIDs (blue) are obtained either by unweighted (wm) or These experiments were run
denotes the weight-
reweighted (wxe) SGM prior training, with only slightly worse on the CIFAR10 dataset, using smaller model than for our main results above (details in App. G).

---

End-to-end training. We proposed to train LSGM end-to-end, in contrast to [10]. Using a similar setup as above we compare end-to-end training of LSGM during the second stage with freezing the VAE encoder and decoder and only training the SGM prior in latent space during the second stage. When training the model end-to-end, we achieve an FID of 5.19 and NELBO of 2.98; when freezing the VAE networks during the second stage, we only get an FID of 9.00 and NELBO of 3.03. These results clearly motivate our end-to-end training strategy.
Mixing Normal and neural score functions. We generally found training LSGM without our proposed "mixed score" formulation (Sec. 3.2) to be unstable during end-to-end training, highlighting its importance. To quantify the contribution of the mixed score parametrization for a stable model, we train small LSGM with only one latent variable group. In this case, without the mixed score, we reached an FID of 34.71 and NELBO of 3.39; with it, we got an FID of 7.60 and NELBO of 3.29. Without the inductive bias provided by the mixed score, learning that the marginal distribution is close to a Normal one for large l purely from samples can be very hard in the high-dimensional latent space, where our diffusion run. Furthermore, due to our importance sampling schemes, we tend to oversample small, rather than large However, synthesizing high-quality images requires an accurate score function estimate for all t. On the other hand, the log-likelihood of samples is highly sensitive to local image statisties and primarily determined at small t. It is plausible that we are still able to learn a reasonable estimate of the score function for these small t even without the mixed score formulation. That may explain why log- ligeelihoood suffers much less than sample quality, as estimated by FID, when we remove the mixed score parameterization.
Additional experiments and model samples are presented in App. H.
6 Conclusions
We proposed the Latent Score-based Generative Model, novel framework for end-to-end training of score-based generative models in the latent space a variational autoencoder. Moving from data to latent space allows us to form more expressive generative models, model non-continuous data, and reduce sampling time using smoother SGMs. To enable training latent SGMs, we made three core contributions: (i) we derived a simple expression for the
entropy term in the variational objective, (ii) we parameterized the SGM prior by mixing Normal and neural score functions, and (iii) we proposed several techniques for variance reduction in the estimation of the training objective. Experimental results show that latent SGMs outperform recent pixel-space SGMs in terms of both data likelihood and sample quality, and they can also be applied to binary datasets. In large image generation, LSGM generates data several orders of magnitude faster than recent SGMs. Nevertheless, LSGM's synthesis speed does not yet permit sampling at interactive rates, and our implementation of LSGM is currently limited to image generation. Therefore, future work includes further accelerating sampling, applying LSGMs to other data types, and designing efficient networks for LSGMs.
7 Broader Impact
Generating high-quality samples while fully covering the data distribution has been a long-standing challenge in generative learning. A solution to this problem will likely help reduce biases in generative models and lead to improving overall representation of minorities in the data distribution. SGMs are perhaps one of the first deep models that excel at both sample quality and distribution coverage. However, the high computational cost of sampling limits their widespread use. Our proposed LSGM reduces the sampling complexity of SGMs by large margin and improves their expressivity further. Thus, in the long term, can enable the usage of SGMs in practical applications.
Here, LSGM is examined on the image generation task which has potential benefits and risks discussed in [94, 95]. However, LSGM can be considered a generic framework that extends SGMs to non-continuous data types. In principle LSGM could be used to model, for example, language [96, 97], music [98, 10], or molecules [99, 100]. Furthermore, like other deep generative models, can potentially be used also for non-generative tasks such as semi- coni-supervised and representation learning [101, 102, 103]. This makes the long-term social impacts of LSGM dependent on the downstream applications.
Funding Statement
All authors were funded by NVIDIA through full-time employment.
10

---

References
[1] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising Diffusion Probabilistic Models. arXiv:2006. //239, 2020.
[2] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations. 2021.
[3] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. Advances Neural Information Processing Systems 32, 2019.
[4] Yang Song. Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. arXiv e-prints, 2021.
[5] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference Learning Representations, 2021.
[6] Prafulla Dhariwal and Alex Nichol, Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.0522 055333,, 2021.
[7] Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating Gradients for Waveform Generation. arXiv:2009.00713, 2020.
[8] Zhifeng Kong, Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. 099666, 2020.
[9] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to- seeteh.. arXiv:2104.00 00499000000,,,0.0000 2021.
[10] Gautam Mittal, Jesse Engel. Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103. 1ane:2003..9991 16091, 2021.
[11] Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-hased generative modeling. In The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online (Palermo, Sicily, Italy]. J. 2020.
[12] Ruojin Cai, Guandao Yang, Hadar Averbuch-Eie Elor, Zekun Hao, Serge J. Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III, volume 12348 of Lecture Notes in Computer Science, pages 364-381. Springer, 2020.
[13] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. 45X15.2103.01455. 2021.
[14] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In The International Conference on Learning Representations (ICLR), 2014.
[15] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approx- imate inference deep generative models. In International Conference on Machine Learning, pages 1278-1286, 2014.
[16] Emilien Dupont, Amaud Doucet, and Yee Whye Teh. Augmented neural ODEs. In Advances Neural Information Processing Systems, pages 3140-3150, 2019.
[17] Zhifeng Kong and Kamalika Chaudhuri. The expressive power of class of normalizing flow models. arXiv preprint (.Xi::000600039 2020.
[18] Chin- Wei Huang, Laurent Dinh, and Aaron Courville. Augmented normalizing flows: Bridging the gap between generative flows and latent variable models. arXiv preprint arXiv:2002.07101 2020.
[19] Jianfei Chen, Cheng Lu, Biqi Chenli, Jun Zhu, and Tian Tian. Vflow: More expressive generative flows with variational data augmentation. arXiv preprint arX(v:2002.09741, 2020.
[20] Arash Vahdat and Jan Kautz. NVAE: A Deep Hierarchical Variational Autoencoder. arXiv:2007.03898, 2020.
[21] Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In International Conference Learning Representations, 2021.

---

[22] Pascal Vincent. A Connection between Score Matching and Denoising Autoencoders. Newral Computa- tion, 2367):1661-1674 2011.
[23] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. arXiv preprint arXiv:/505.057755 2015.
[24] Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pages 4743-4751, 2016.
[25] Chin-Wei Huang, Jae Hyun Lim, and Aaron Courville. A variational perspective on diffusion-based generative models and score matching. arXiv preprint arXiv:2106. 028008 2021.
[26] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. arXiv preprint arXiv:2102.09672, 2021.
[27] Jascha Sohl- Dicks-iikntenn Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning Using Nonequilibrium Thermodynamics. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICMI. 15, page 2256-2265. JorR.org. 2015.
[28] c. Monte Carlo theory, methods and examples. 2013.
[29] Aapo Hyvärinen. Estimation of Non-Normalized Statistical Models by Score Matching. J. Mach. Learn. Res., 6:695-709, December 2005.
[30] Siwci Lyu. Interpretation and Generalization of Score Matching. In Proceedings of the Twenty- Conference Uncertainty in Artificial Intelligence, UAI '09, page 2009. AUAI Press.
Arlington, Virginia, USA,
[31] Durk P Kingma and Yann I.. Cun. Regularized estimation of image statistics by Score Matching. In Advances in Neural Information Processing Systems 23, 2010.
[32] Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized Denoising Auto-Encoders as Generative Models. In Proceedings of the 26th International Conference on Neural Information Processing Systems, 2013.
[33] Krzysztof J. Geras and Charles A. Sutton. Scheduled denoising autocncoders. In 3rd International Conference on Learning Representations, ICLR, 2015.
[34] Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, and Aapo Hyvärinen. Deep Energy Estimator Networks. 1805 2018
[35] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced Score Matching: A Scalable Approach Density and Score Estimation. Proceedings of the Thirty-Fifth Conference on Uncertainty Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, 2019.
[36] Zengyi Li, Yubei Chen, and Friedrich T. Sommer. Learning Energy- Based Models in High-Dimensional Spaces with Multi-scale Denoising Score Matching. /91.07911. 2019.
[37] Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient Learning of Generative Models via Diffe--Difeeence Score Matching, arXiv:2007.03317, 2020.
[38] Yang Song and Stefano Ermon. Improved Techniques for Training Score- Base- Generative Models. arXiv:2006.0901. 2020.
[39] Alexia Jolicoeur-Martincau. Rémi Fiché-Taillefen Ioannis Mitliagkas, and Remi Tachet des Combes. Adversarial score matching and improved sampling for image generation. In International Conference on Learning Representations, 2021.
[40] Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, Diederik Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations, 2021.
[41] Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprini arXiv:2101.02555 2021.
[42] Robin San-Roman Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion models. arXiv preprint 2rXiv:2104.02600, 2021.
12

---

[43] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. arXiv: 03480, 2018
[44] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Proceedings of the 35th International Conference Machine Learning, 2018.
[45] Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[46] Renjic Liao, Yujia Li, Yang Song, Shenlong Wang. Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems, 2019.
[47] Xi Chen, Diederik Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Picter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv: 1,02731, 2016.
[48] Lars Maalge, Marco Fracearo, Valentin Liévin, and Ole Winther. BIVA: A very deep hierarchy of latent variables for generative modeling. In Advances in neural information processing systems, pages 2019.
[49]
Jason Tyler Rolfe Discrete variational autoencoders. arXiv preprint arXiv: 2016.
[50] Arash Vahdat, Evgeny Andriyash, and William G Macready. DVAE#: Discrete variational autoencoders with relaxed Boltzmann priors. Neural Information Prucessing Systems, 2018.
[51] Arash Vahdat, William G. Macready, Zhenghing Bian, Amir Khoshaman, and Exgeny Andriyash. DVAF+ Diserete variational autoencoders with overlapping transformations. In International Confer- ence on Machine Learning (ICMI.), 2018.
[52] Arash Vahdat, Evgeny Andriyash, and William G Macready. Undirected graphical models approximate posteriors. International Conference on Machine Learning (ICML), 2020.
[53] Bo Pang. Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy- based prior model. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 21994-22000 Curran Associates, Inc., 2020.
[54] Jesse Engel, Matthew Hoffman, and Adam Roberts Latent constraints: Learning generate conditionally from unconditional generative models. International Conference on Learning Representations, 2018.
[55] Matthias Bauer and Andriy Mnih. Resampled priors for variational autoencoders. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 66-75.
PMLR, 16-18 Apr 2019.
[56] Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. NCP-VAE: Variational autoencoders with noise contrastive priors. arXiv preprint arXiv:2010.02917 2020.
[57] Alireza Makhzani, Jonathon Shiens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders, 2016.
[58] Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Variational autoeneoder with implicit optimal priors. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01))5066-500 5073, Jul. 2019.
[59] Jakuh Tomezak and Max Welling. Vae with vampprior. In International Conference on Artificial Intelligence and Statistics, pages 1214-1223, 2018
[60] Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
[61] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXiv:1711 1701.00977. 2018.
[62] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2.
In Advances in Neural Information Processing Systems, pages 14837-14847, 2019.
[63] Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations. 2020.
13

---

[64] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. arXiv preprint arXiv:2012.09841, 2020.
[65] Abdul Fatir Ansari, Ming Liang Ang, and Harold Soh. Refining deep generative models via discriminator gradient flow. International Conference on Learning Representations, 2021.
[66] Tong Che, Ruixiang ZHANG, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull. Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-basee based model and you should use discriminator driven latent sampling. In Advances in Neural Information Processing Systems, 2020.
[67] Akinori Tanaka. Discriminator optimal transport. In Advances in Neural Information Processing Systems, 2019.
[68] Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with latent-space energy-based models. In Neural Information Processing Systems (NeurIPS), 2021.
[69] Antoine Wchenkel and Gilles Louppe. Diffusion priors in variational autocncoders. In ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2021.
[70] Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-denoising models for few-shot conditional generation. arXiv preprint arXiv:2106.068 06819, 2021.
[71] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, arXiv preprint arXiv:2107.000 2021.
[72] Dongjun Kim, Seungjae Shin, Kyungwoo Song, Wanmo Kang, and II-Chul Moon. Score matching model for unbounded data score. arXiv preprint wxXv:2206.05557, 2021.
[73] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018.
[74] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv: 1509:000000000 1509,0059, 2015.
[75] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.
[76] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021.
[77] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinetive generative autoencoder. arXiv preprint arXiv:201 10063, 2020.
[78] Heewoo Jun, Rewon Child, Mark Chen, John Schulman, Aditya Ramesh, Alec Radford, and llya Sutskever. Distribution augmentation generative modeling. In International Conference on Machine Learning. pages 5006-5019. PMLR, 2020.
[79] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904 10509, 2019.
[80] Ali Razavi, Aäron van den Oord, Ben Poole, and Oriol Vinyals. Preventing posterior collapse with delta-vaes. In The International Conference on Learning Representations (ICLR), 2019.
[81] XI Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixeISNAIL: An improved autoregres- sive generative model. In International Conference on Machine Learning, 2018.
[82] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixeICNN+:: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. arXiv preprint urXiv:1701.05517 2017.
[83] Xinyu Gong, Shiyu Chang, Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2019.
[84] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila, Training generative adversarial networks with limited data. In NewrlPS, 2020.
14

---

[85] Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1xl convolutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems 31, pages 10236-10245. 2018.
[86] Jacoh Menick and Nal Kalchhrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. International Conference on Learning Representations. 2019.
[87] Stanislav Pidhorskyi. Donald Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[88] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing GANs for improved quality, stability, and variation. In International Conference on Learning Representations, 2018.
[89] Alex Krizhevsky et al. Learning multiple layers of features from tiny images, 2009.
[90] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pages 3738-3746, 2016.
[91] Hossein Sadeghi, Evgeny Andriyash, Walter Vinci, Lorenzo Buffoni, and Mohammad Amin. Pixel- vae++: Improved pixelvae with discrete prior. arXiv preprint
2019.
[92] Xuezhe Ma, Chunting Zhou, and Eduard Hovy. MAE: Mutual posterior-divergence regularization for variational autoencoders. In The International Conference on Learning Representations (ICLR), 2019.
[93] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. comlexdh/mnist/, 1998.
[94] J. Bailey, The tools of generative art, from flash neural networks. America, 2020.
[95] Cristian Vaccari and Andrew Chadwick. Deepfakes and disinformation: Exploring the impact synthetic political video on deception, uncertainty, trust news. Social Media+ Society. 6(1):2056305 120903408, 2020.
[96] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a contimuous space. In Procedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-2) (1-21,22 2016.
[97] Chumyuan Li, Xiang Gao, Yuan Li, Xiujun Li, Baolin Peng, Yizhe Zhang, and Jianfong Gao. Optimus: Organizing sentences via pre-trained modeling of a latent space. In EMNLP. 2020.
[98] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00:4 2020.
[99] Benjamin Sanchez-Lengeling and Alán Aspuru-Guzik. Inverse molecular design using machine learning: Generative models for matter engineering. Science, (6400)))33 365, 2018.
[100] Zaccary Alperstein, Artem Cherkasov, and Jason Tyler Rolfe. All smiles variational autoencoder. arXiv preprint 1rXiv:1905.13342 2019.
[101] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pages 3581-3589 2014.
[102] Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint 1rXiv:1606.01583, 2016.
[103] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralha, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-n--dommin generalization. In Conference on Computer Vision and Pattern Recognition (CVPR). 2021.
[104) Simo Särkkä and Arno Solin. Applied Stochastic Differential Equations. Institute of Mathematical Statistics Textbooks. Cambridge University Press, United Kingdom, 2019.
[105] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Process. Appl., 12(3):313-326, 1982.
[106] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Fijord: Free-form continuous dynamics for scalable reversible generative models. International Conference on Learning Representations, 2019.
15

---

[107] Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019.
[108] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[109] J. R. Dormand and J. Prince. A family of embedded Runge-Kutta formulae. Journal of Computationa and Applied Mathematicx, 6(1):19-26, 1980.
16


Reviews:
[{"review_id": "msc7Hr9v95V", "reviewer": "Reviewer_N7bW", "summary": "The paper proposes to learn a flexible VAE prior using score-based generative model. Contrary to the existing score-based model that builds on high-dim pixel space, the paper applies the score-based model directly on the latent space which is typically low-dimensional.  The effectiveness of the model has been verified using generative FID score, negative log-likelihood as well as various ablation studies. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n\n (+/-) The model presented is relatively intuitive and seems not appeared in the recent literature. However, the idea that uses some recently well-performed diffusion/score-based model as a flexible prior of VAE is straightforward and appears less surprising. \n\nQuality: \n\n (+) The authors put efforts on designing various ablation studies to explore their model and conduct quite a lot comparisons with existing baselines. The overall generative performance looks good. I also appreciate the authors provide comprehensive supplementary materials for additional proof, details and results. \n\n (-) Though the FID score is good, but the vae and Sgm backbone structure can be complicated, and some places need better described in the main text. I understand the powerful backbone structures are used to target SOTA performance. But from the model/methodology side, it might be better to also focus on the \"basic\" backbone structures on which quite a few existing generative models build (e.g., the vae backbone with a few convolution-relu layers). Does the paper try such basic backbone models in the experiments? In the tab 2& 3, Ours (VAE backbone) refers to the non-NVAE structure? What structure exactly? Also, does all the results obtained by pre-training the NVAE first OR jointly train both VAE and SGM? Would be more interesting if the latter.  I would adjust my initial scores based on the feedback of the authors on above issues. \n\nClarity: \n \n(+) overall, the paper is easy to follow and presentation is clear despite a few places about training procedures as listed above. \n\nSignificance: \n\n (+/-) The idea is intuitive and could be easy to use/adapt for the follow-up works, but the idea itself is not so surprising, and cannot be considered transformative.\n \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper mentions its limitation and broader impact. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "f9NE9qDdwGX", "reviewer": "Reviewer_ipL2", "summary": "The paper at hand proposes end-to-end training of score-based generative models in the latent space of a variational autoencoder. The VAE is pre-trained using a normal prior which is then replaced by the score-based generative model that is then jointly trained with the VAE. The paper also introduces a novel training objective based on the cross entropy between the encoder distribution and the SGM prior. Two techniques for variance reduction of the loss function are discussed.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "REASONS FOR SCORE:\n\n- The idea of applying the score based model to the latent space of a VAE is novel, as far as I can see.\n- The experimental evaluation yields promising results on 5-bit CIFAR10, CelebA and binarized MNIST and OMNIGLOT: the proposed LSGM outperforms a variety of generative models in terms of test likelihood and FID-score (although only by a very small margin). The value of the proposed model lies within the sampling time which is decreased from 44.6 min to to 3.91 minutes for a batch of 16 images, which is remarkable.\n- The choice of models that are compared is, in my opinion, complete and the experimental evaluation is overall very convincing. Only [1] achieve slightly better results on CelebA, however the paper appeared on arxiv after the NeurIPS deadline. In Table 3 the authors report an FID score of 10.70 on CelebA for [2], however in Table 3 of [2] an FID of 10.2 is reported (again this last version was uploaded after the deadline). \n- The structure of the paper is reasonable. Although the paper is very technical, the authors succeeded at clearly explaining their approach. A short background section introduces the relevant concepts for readers that are not familiar with score-based generative models. During reading the paper, I spotted no typos.\n- The authors justify their choices by an ablation study where they analyze the effect of SDEs, training objectives, weighting mechanisms, and variance reduction techniques. As a small point of criticism: I had a hard time understanding Table 6. The authors should spend some time refactoring the table and its very long caption.\n\nCONCLUSION:\n\nOverall, I would reccomend to accept this submission into NeurIPS 2021. The paper is well-written and addresses the efficiency issues of score-based generative models. The experimental results on the benchmark datasets are promissing.\n\nMINOR REMARKS:\n\n- Paragraph captions not always in title-case (for example: \"Implementation details\")\n\n\nREFERENCES:\n\n[1] Kim, Dongjun, et al. \"Score Matching Model for Unbounded Data Score.\" arXiv preprint arXiv:2106.05527 (2021).\n\n[2] Esser, Patrick, Robin Rombach, and Bjorn Ommer. \"Taming transformers for high-resolution image synthesis.\" arXiv preprint arXiv:2012.09841 (2021)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "ZHDyQ5WYphl", "reviewer": "Reviewer_2Ubw", "summary": "The paper proposes a generative model based on the VAE framework with a sophisticated prior / inference scheme based on denoising score-matching. Authors develop extensive machinery to adapt related ideas from score-based generation in the observed space to latent space. This greatly improves sampling time while preserving and sometimes improving sample quality and/or likelihood of data. I believe the paper is an interesting addition to the existing collection of generative modelling techniques.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\n\nTo my knowledge, the method is novel. There is a certain intersection with some concurrent NeurIPS submissions though which will can be reflected by authors when preparing the final version of the paper.\n\nClarity:\n\nThe paper is generally well-written but due a large amount of mathematical material, the writing is dense and there are some details that I would to be discussed more. \n\n1) As I understand, the exceptionally good results with modelling binary images are explained by the fact that score-matching in the latent space resulted into very tight variational approximation and, hence, good gradients for training the decoder. Would it be possible to quantify this directly by measuring the KL divergence for the LSGM and the same model trained with standard amortized inference techniques? Or there are even better ways to further illustrate this?\n2) I couldn't find detailed information on how the structure of the latent space. As I understand, it's a lower resolution image-like structure. If so, can authors visualize evolution of the latent variable under the SDE and what reconstructions does it produce? \n3) Clearly, effectiveness of LSGMs depends on the choice of the latent space, and I would like to get, again, a more detailed discussion around this. How do the performance metrics depend on its size and structure (image-like vs flat vector)?\n\nQuality:\n\nI didn't read proofs for the presented theorems, but I followed equations in the main text and they made sense to me.\n\nSignificance:\n\nI believe LSGMs are a valuable contribution because it can enable faster generation and tighter variational approximations.\n\nI think the potential impact can be further improved if similarly better results are obtained with decoder architectures othen than NVAE which is arguably very specially structured in handling latent variables.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No issues.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Rm-yB2bgkfN", "reviewer": "Reviewer_Gyo1", "summary": "The paper proposes Latent Score-based Generative Model (LSGM) which introduces a score-based prior in the Variational Autoencoder (VAE) framework. In contrast to previous score-based models that operate in the data space, LSGM uses a score-based model in the latent space. A sample from a base distribution (standard Gaussian) is denoised using the score-based prior in the latent space and is then mapped to the data space using a decoder. The authors further discuss how the various terms in the ELBO can be computed to train the model in an end-to-end fashion without requiring the time-dependent marginal score function. Multiple variance reduction techniques and training tricks have also been proposed for the resulting objective function. Both quantitative and qualitative results demonstrate the ability of the model to generate high fidelity images.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Significance\n\nThis paper makes a significant contribution both to the fields of score-based models and VAEs with more expressive priors. Severals well-motivated training stabilization techniques have been proposed which may prove helpful for the community. The paper also improves the state of the art in terms of sample quality (FID) for VAEs.\n\n### Clarity\n\nThe paper is well-written and organized. There are a fews places where inclusion of details in the main text would improve the clarity of the paper:\n\ni) Line 148: The parameterization of score function using $\\epsilon_\\theta$.    \nii) Line 263: Differences between the three models.\n\n### Technical quality\n\nThe paper is technically sound. The empirical results are fairly comprehensive and evaluate LSGM on various datasets against state-of-the-art deep generative models. Ablation studies presented help elucidate the contribution of different components of the proposed model.\n\n### Originality/Relation to prior work\n\nThe paper presents a novel combination of score-based modeling with VAEs: a score-based prior distribution is introduced in the VAE framework. The relation to prior-work has been discussed sufficiently. \n\nThe proposed work is also related to recent works [e.g., 1, 2, 3] that propose techniques to improve samples from deep generative models (particularly GANs). In these works, latent vectors are first diffused to a \"better\" point and then decoded to generate a sample. This is similar to LSGM where the latent vectors are denoised using a score-based model. A discussion of relation to these works would improve the related works section.\n\n[1] Tanaka, Akinori. \"Discriminator optimal transport.\" arXiv preprint arXiv:1910.06832 (2019).\n\n[2] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. \"Refining deep generative models via discriminator gradient flow.\" arXiv preprint arXiv:2012.00780 (2020).\n\n[3] Che, Tong, et al. \"Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling.\" arXiv preprint arXiv:2003.06060 (2020).\n\n### Additional comments/Questions\n\n- Did the authors investigate what final value of $\\alpha$ is learned by the model? Specifically, is there an \"optimal\" mixing coefficient?\n- In the ablation study the authors mention that the mixture formulation is particularly important for the sample quality. Do the authors have any insights on why this is the case?\n\n-------\nPost Rebuttal: Thank you for the clarification on $\\alpha$ and the mixture formulation. I believe that a detailed discussion of the mixture formulation will improve the manuscript. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that this was a novel and exciting contribution. The author-reviewer discussion further supported the paper's quality.
Decision: Accept (Poster)