D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation
Authors: Abhishek Sinha
Jiaming Song
Chenlin Meng
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative models

Abstract: Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAE) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks, conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study. We release our code at https://github.com/jiamings/d2c.

D2C: Diffusion-Decoding Models for Few-Shot Conditional Generation
Abhishek Sinha" Department of Computer Science Stanford University a7 37b23@stanford.. edu
Jiaming Song" Department of Computer Science Stanford University tsongâcs stanfsrafrrroo edu
Chenlin Meng Department of Computer Science Stanford University chenlin@ce stanford. edu
Stefano Ermon
Department Computer Science
Stanford University ermonêcs stanford edu
Abstract
Conditional generative models of nigh-dimensiooal images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decodiin models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses learned diffusion-based prior over the latent representations to improve generation and contrastive self- supervised learning to improve representation quality. D2C can adapt novel generation tasks conditioned on labels or manipulation constraints, by learning from as few 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% 60% of the human evaluators in a double-blind study. We release our code at https: //github com/ iamings/d2c.
Introduction
Generative models trained on large amounts unlabeled data have achieved great success in various domains including images [8, 50, 75, 42], text [56, 2], audio [26, 71, 93, 62], and graphs [36, 67]. However, downstream applications of generative models are often based on various conditioning signals, such labels [6]], text descriptions [60], reward values [101], similarity with existing data [45]. While it is possible to directly train conditional models, this often requires large amounts of paired data [57, 74] that are costly to acquire. Hence, would be desirable to learn conditional generative models using large amounts of unlabeled data and as little paired data as possible.
Contrastive supevvupe learning (SSL.) methods greatly reduce the need for labeled data in discriminative tasks by learning effective representations from unlabeled data [95, 37, 35], and have also been shown improve few-shot learning [39]. It is therefore natural ask if they be used to improve few-shot generation. Latent variable generative models (LVGM) are a natural candidate for this, since they already involve tow-dimensional, structured latent representation of the data they generate. However, generative adversarial networks (GANs, [34, 50]) and diffusion
also
Equal contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

models [42, 84], lack explicit tractable functions to map inputs to representations, making it difficult optimize latent variables with SSL. Variational autoencoders (VAEs, [52, 77]), on the other hand, can naturally adopt SSI. through their encoder model, but they typically have worse sample quality.
Z
X
A
"blond hair"
(z(2)
iPe (x|z,
z
r=(c/2)
+ "red lipstick"
X
c
"Blond hair"
Unconditional model from unlabeled data
Recognition model from few-shot supervision
Conditional generation
Figure 1: Few-shot conditional generation with the unconditional D2C model (left). With recogni- tion model over the latent space (middle), D2C can generate samples for novel conditions, such as image manipulation (right). These conditions can be defined with very few labels.
In this paper, we propose Diffusion-Decodeee models with Contrastive representations (D2C), special VAE that is suitable for conditional few-shot generation. D2C uses contrastive self-supervised learning methods to obtain latent space that inherits the transferrability and few-shot capabilities of self-supervised representations. Unlike other VAEs, D2C learns a diffusion model over the latent representations. This latent diffusion model ensures that D2C uses the same latent distribution for both training and generation. We provide a formal argument to explain why this approach leads to better sample quality than existing hierarchical VAEs. We further discuss how apply D2C to few-shot conditional generation where the conditions are defined through labeled examples or manipulation constraints. Our approach combines discriminative model providing conditioning signal and generative diffusion model over the latent space, and is computationally more efficient than methods that act directly over the image space (Figure) 1).
We evaluate and compare D2C with several state-of-the-an generative models over datasets. On unconditional generation, D2C outperforms state-of-the-art VAEs and is competitive with diffusion models under similar computational budgets. On conditional generation with 100 labeled examples, D2C significantly outperforms oftte-of-hhearr VAE [91] and diffusion models [84]. D2C can also learn to perform certain image manipulation tasks from as few as 100 labeled examples. Notably, for manipulating images, D2C is two orders of magnitude faster than StyleGAN2 [106] and preferred by 50% 60% of human evaluations, which to our best knowledge is the first for any VAE model.
2
Background
Latent variable generative models latent variable generative model (LVGM) posed as con- ditional distribution pe 2 P(X) from a latent variable z to generated sample parametrized by 6. To acquire new samples, LVGMs draw random latent variables z from some distribution p(z) and map them to image samples through x po (x(). Most LVGMs are built on top of four paradigms: variational autoencoders (VAFs, [52, 77]), Normalizing Flows (NFs, [28, 29]), Generative Adversarial Networks (GANs, [34]), and diffusion score-based generative models [42, 85).
Particularly, VAES an inference model from z for training. Denoting the inference distribution from x to z as (z) x), the generative distribution from z to x as pa (x(z)), VAEs are trained by minimizing the following upper bound negative likelioh:hood::
LAKE = Ego-patta Ez (xlx))()))) logp(x(2)) + (
(1)
where Pdata is the data distribution and DKL is the KL-divergence.

---

Diffusion models Diffusion models [81, 42, 84] produce samples by reversing Gaussian diffusion process. We use the index E [0, 1] to denote the particular noise level of an noisy observation x(*) = vax + v1 ac, where x is the clean observation and c ~ (O, I) a standard Gaussian distribution; as a - O, the distribution of x(o) converges to (0, I). Diffusion models are typically parametrized as reverse noise models eg (x((("), a) that predict the noise component of x(() given a noise level a, and trained to minimize []co(x(a), a) - <lβ- the mean squared error loss between the true noise and predicted noise. Given any non-increasing series {or?? between 0 and 1, the diffusion objective for clean sample from the data is:
T w E(a -e (x( aa) a.)l x(ax) == Va,x + VI age (2)
lant(x w, 0) =
Σ
where w fai)e R.+ controls the loss weights for each When w(a) for all a, recover the denoising score matching objective for training score-based generative models [85].
Given an initial sample Xo (0, 1), diffusion models acquires clean samples (i.e., samples of x1) through a gradual denoising process, where samples with reducing noise levels a are produced (e.g-- Xo -> Xo.3 + Xo.7 + X1). In particular, Denoising Diffusion Implicit Models (DDIMs, [84]) uses an Euler discretization of some neural ODE [13] to produce samples (Figure 2, left).
We provide a more detailed description for training diffusion models in Appendix A.1 and sampling from DDIM in Appendix A.2. For conciseness, we use the notation pia) (x(()) to denote the marginal distribution of x!a) under the diffusion model, and p ((01:02) (x((2)) (0x) (x())) to denote the diffusion
sampling process from (1) to x((2)) (assuming 011 < a:2).
Self-supervised learning of representations In self self-supervised learning (SSL), representations are learned by completing certain pretext tasks that do not require extra manual labeling [68, 25]; these representations then be applied to other downstream tasks, often in few-shot or zero-shot scenarios. In particular, contrastive representation learning encourages representations to be closer between "positive" pairs and further between "negative" pairs; contrastive predictive coding (CPC, [95], based on multi-class classification, have been commonly used in state-of-the-arr methods [37, 16, 18, 14, 82].
Other non-contrastive methods exist, such as B BOL [35] and SimSiam [17], but they usually require additional care to prevent the representation network from collapsing.
3 Problem Statement
Few-shot conditional generation Our goal is to learn an unconditional generative model Pa such that it is suitable for conditional generation. Let C(x, c, f) describe an event that "f(x) c". where c is a property value and fxx) is a property function that is unknown at training. In conditional generation, our goal is to sample x such that the event C(x, c, f) occurs for chosen If we have access to some "ground-truth" model that gives us p[C]x) := fff(x)= = cix). then the conditional model can be derived from Bayes' rule: po (x(C) x p(C]x)pp (x). These properties include (but not limited to²) labels [61], text descriptions [60, 76], noisy or partial observations [11, 5, 47, 24], and manipulation constraints [69]. In many cases, we do not have direct access to the true fxx). so we need to learn an accurate model from labeled data [6] (e.g.,
S. (c, x) pairs).
Desiderata Many existing methods are optimized for some known condition (e.g., labels in con- ditional GANs [8] or assume abundant pairs between images and conditions that can be used for pretraining (e.g., DALL-E [74] and CLIP [73] over image-text pairs). Neither is the case in this paper, as we do not expect to train over paired data.
While high-quality latent representations are not essential to unconditional image generation (e.g., autoregressive [94], energy-based [31], and some diffusion models [42]), they can be beneficial when wish specify certain conditions with limited supervision signals, similar to how SSL representations can reduce labeling efforts in downstream tasks. A compelling use case is detecting and removing biases in datasets via image manipulation, where we should not only address well- known biases a-priori but also address other hard-to-anticipaa biases, adapting to societal needs [65].
When C refers to event that is always true, recover unconditioned generation.
3

---

Table comparison several common paradigms for generative modeling. [Explicit x z]: the mapping from x to is directly trainable, which enables SSL; [No prior hole]: latent distributions used for generation and training are identical (Sec. 4.2), which improves generation; - training procedure does not involve adversarial optimization, which improves training stability.
Explicit - 2 (Enables SSL)
No prior hole (Better generation)
Non-Adversarial (Stable training)
Model family
VAE [52, 77], NF [28]
GAN [34]
x
X
BiGAN [30
X
DDIM [84]
X
D2C
Therefore, a desirable generative model should not only have high sample quality but also contain informative latent representations. While VAFs are ideal for learning rich latent representations due to being able to incorporate SSL. within the encoder, they generally do not achieve the same level of sample quality GANs and diffusion models.
4 Diffusion-Decoding Generative Models with Contrastive Learning
In this paper, present Inffusiion-Dcodiii generative models with Contrastive Learning (D2C), an extension to VAEs with high-quality samples and high-quality latent representations, and are thus well suited to few-shot conditional generation. Moreover, unlike GAN-based methods, D2C does not involve unstable adversarial training (Table 1).
As its name suggests, the generative model for D2C has two components diffusion and decoding; the diffusion component operates over the latent space and the decoding component maps from latent representations to images. Let use the index notation for diffusion random variables: (( ~ (0) (z( := NN I) is the "noisy" latent variable with O. and (1 is the "clean" latent variable with 1. The generative process of D2C, which we denote (x (")),, is then defined as:
(0) ~ p (z) 2(1) Po (0.1) (0))
X ~ Pa (x
(3)
diffusion
decoding
where (0)) (2)0) = =N is the prior distribution for the diffusion model, Pe (0.1) (z (0) is the diffusion process from z (0) (1)), and po ()) is the decoder from (1)) to x. Intuitively, D2C models produce samples by drawing z(1) from a diffusion process and then decoding x from (1).
In order to train D2C model, we use inference model 4s (z))) |x) that predicts proper (1)) latent variables from x; this can directly incorporate SSL methods [99], leading to the following objective:
TDp2c (f, d; w) LD2200, w) XLc(go),
(02 d; w) = Ex mas (2(1) 1- (x (x(z( (zir((z (1) w, 0)},
(5)
where fai is defined as in (2), Le (qa) denotes any contrastive predictive coding objective [95] with rich data augmentations [37, 16, 18, 14, 82] (details in Appendix A.3) and À > is a weight hyperparameter. We illustrate D2C in Figure 2, and its training procedure in Appendix A.4.
4.1 Relationship to maximum likelihood
The D2 objective (LD2) appears similar to the original VAE objective (LVAE). Here, make an informal statement that the D2 objective function is deeply connected to the variational lower bound of likeliie;;hoo;; present the full statement and proof in Appendix B.I.
Theorem 1. (informal) For any valid {ax }T there exists some weights úi [a:)T= - Th for the diffusion objective such that - Lpz is a variational lower bound to the log-likelihood, iee.
-Dp (0, 6; u) & Epas [log pe (x)],
(6)
where (x = E (2(0)) [x (x|z ())] is the marginal probability of x under the model.

---

x
Diffusion-bald baaed prior: superior sample quality
1x(2(1/x)
(z) (
z (2)
Contrastive encoder: better few-shot capabilities
A6 (x
(x
Decoder pe(x|2
x
Figure 2: Illustration of components of D2C model. top of the encoding and decoding between
and z(1, we use diffusion model generate z(1) from Gaussian ())) The red lines in the diffusion prior describe several smooth ODE trajectories from a 0 to 1.
Proof. (sketch) The diffusion term Lai upper bounds the KL divergence between g and Pg (x()) for suitable weights [42, 84], which recovers a VAE objective.
Moreover, since be treated as an implementation of the InfoVAE approach [103], where the latent variables are encouraged to have higher mutual information with the observations. Since the evidence lower bound (ELBO) for VAEs do not necessarily promote informative latent representations [15], D2C comes with the benefit of being able learn more informative latents despite not exactly optimizing an ELBO.
contrastive losses lower bounds mutual information [82], D2C can
4.2 D2C models address latent posterior mismatch in VAEs
While D2C is special case of VAE, we argue that D2C is non-trivial in the sense that it addresses long-standing problem in VAE methods [90, 87], namely the mismatch between the prior distribution pz(z) and the aggregate (approximate) posterior distribution (z) == ( match could create "holes" [79, 43, 3] in the prior that the aggregate posterior fails to cover during training, resulting in worse sample quality, as many latent variables used during generation are likely to never have been trained on. We formalize this notion in the following definition.
(x) 1g (z x)]. A mis-
Definition (Prior hole). Let p(z), ((z) be two distributions with supp(g) S supp(p). We say that q has an (e, -)prio hole with respect to (the prior) p for e, 10. d if there exists a set S (P) such that Ig p(z) plz)dz> 2 and Ig gz)) S.
Intuitively, if ga (z) has a prior hole with large and small (e.g., inversely proportional to the number of training samples), then it is very likely that latent variables within the hole are never seen during training (small c), yet frequently used to produce samples (large 8). Most existing methods address this problem by optimizing certain statistical divergences between (z(z) and po (z), such as the KL divergence or Wasserstein distance [88]. However, we argue in the following statement that prior holes might not be eliminated even if we optimize certain divergence values to be reasonably low, especially when (z) is very flexible. We present the formal statement and proof in Appendix B.2.
Theorem 2. (informal) Let pz(z) N(O, 1). For any O. there exists (z) with an (c, 0.49))prior hole, such that DRL. (gs llpo) S log 2 and W2(qs, Po) < for any 7 > 0, where W2 the 2- 2-Wasserstein distance.
This is reasonably low for realistic VAE models (NVAE [91] reports KL divergence of around 2810 nats).
5

---

Proof. (sketch) We construct that satisfies these properties (top-right figure). First, we truncate the Gaussian and divide them into regions with same probability mass; then we support gs over half of these regions (so å >> 0.49); finally, we show that the divergences are small enough.
In contrast to addressing prior holes by optimization, diffusion models eliminate prior holes by construction, since the diffusion process from (2) to z is constructed such that the distribution of 2()) always converges to a standard Gaussian as a + 0. As a result, the distribution of latent variables used during training is arbitrarily close to that used in generation', which is also the case in GANs. Therefore, our argument provides explanation as to why we observe better sample quality results from GANs and diffusion models than VAEs and NFs.
5 Few-shot Conditional Generation with D2C
In this section, we discuss how D2C can be used to learn to perform conditional generation from few-shot supervision. We note that D2C is only trained on images and not with any other data modalities (e.g., image-text pairs [74]) or supervision techniques (e.g., meta-learning [22, 6D.
Algorithm We describe the general algorithm for conditional generation from a few images
Algorithm 1 Conditional generation with D2C
I: Input examples ((x;, c;)) property
in Algorithm I, and detailed implementations in Appendix C. With model over the latent
2: Acquire latents x!" ~ qo (x(2"))xx for i E [n]]
space (denoted as ry (c), we draw condi- tional latents from an unnormalized distribution
3: Train model ra (clz(()) over ((al), ,c:)]==
4: Sample latents with (1) ro(cc//))) .pp (p (x(") (unnormalized))
with the diffusion prior (line 4). This can be implemented in many ways such as rejection sampling or Langevin dynamics [66, 86, 27].
5: Decode x ~ ())
6: Output &
Conditions from labeled examples Given a few labeled examples, we wish to produce diverse samples with certain label. For labeled examples we directly train classifier over the latent space, which we denote as re (c/z(1) with e being the class label and (11) being the latent representation of from qs (z)) xx). If these examples do not have labels (i.e., we merely want to generate new samples similar to given ones), we can train a positive-unlabeled (PU) classifier [33] where assign "positive" to the new examples and "unlabeled" to training data. Then we use the classifier with the diffusion model po (2(1) ())) to produce suitable values of 2 such as by rejecting samples from the diffusion model that has a small n. (c)z
Conditions from manipulation constraints Given a few labeled examples, here we wish to learn how manipulate images. Specifically, we condition over the event that "x has label e but is similar to image x" Here (1 is the unnormalized product between the classifier conditional probability and closeness the latent (11) of x (e.g., measured with RBF kernel). We implement line 4 of Alg. with a I Lanvegin-like procedure where we take a gradient step with respect to the classifier probability and then correct this gradient step with the diffusion model. Unlike many GAN-based methods [12, 72, 97, 45, 98], D2C does not need to optimize an inversion procedure at evaluation time, and thus the latent value much faster compute; D2C is also better at retaining fine-grained features of the original image due to the reconstruction loss.
6 Related Work
Latent variable generative models Most deep generative models explicitly define a latent rep- resentation, except for some energy-based models [41, 31] and autoregressive models [94, 93, 10]. Unlike VAEs and NFs, GANs do not explicitly define an inference model and instead optimize a two-player game. In terms of sample quality, GANs currently achieve superior performance over VAFS and NFs, but they can be difficult to invert even with additional optimization [48, 100, 7]. This can be partially addressed by training reconstruction-base losses with GANs [54, 55J. Moreover,
We expand this argument in Appendix B.2.

---

the GAN training procedure can be unstable [9, 8, 63], lack a informative objective for measuring
progress [4], and struggle with discrete data [102]. Diffusion models [27] achieves high sample quality without adversarial training, but its latent dimension must be equal to the image dimension. A concurrent work [92] applied score-based generative modeling on the latent space of an VAE, and achieved similar improvements compared to regular VAEs.
Addressing posterior mismatch in VAEs Most methods address this mismatch problem by im- proving inference models [64, 51, 89], prior models [90, 3, 87], or objective functions [103, 104, 105, I, 59]: all these approaches optimize the posterior model to be close to the prior. Section 4.2, explain why these approaches do not necessarily remove large "prior holes", their sample qualities remain relatively poor even after many layers [91, 19]. Other methods adopt a "two-stage" approach [23], which fits generative model over the latent space of autoencoders [96, 75, 26, 74). [62] have applied diffusion models directly on latent spaces learned by a music VAE.
Conditional generation with unconditional models To perform conditional generation unconditional IVGM, most methods assume access to discriminative model (e.g., a classifier); the latent space of the generator is then modified to change the outputs of the discriminative model. The disciminative model can operate on either the image space [66, 70, 27] or the latent space [80, 98]. For image space discriminative models, plug-and-play generative networks [66] control the attributes of generated images via Langevin dynamics [78]; these ideas are also explored in diffusion models [86]. Image manipulation methods are based on GANs often operate with latent space diseriminators [80, 98]. However, these methods have some trouble manipulating real images because of imperfect reconstruction [107, 7]. This is not problem in D2C since a reconstruction objective optimized.
Experiments
We examine the conditional and unconditional generation qualities of D2C over CIFAR-10 [53], CIFAR-100 [53], fMoW [21], CelebA-64 [58], CelebA-HQ-256 [48], and FFHQ-256 [49]. Our D2C implementation is based on the state-of-the-art NVAE [91] autoencoder structure, the U-Net diffusion model [42], and the MoCo-v2 contrastive representation learning method [16]. We also consider the D)2 objective where we do not applying the contrastive loss in the NVAE autoencoder; we also tried adding contrastive loss directly to the VAE objective. but we were unable to achieve satisfactory generation results (reconstruction MSE remains high). This possibly due the many regularizations needed for NVAE to work well, which could conflict with contrastive learning'.
We keep the diffusion series hyperparameter {ar]T= identical to ensure fair comparison with different diffusion models. For the contrastive weight À in Equation (4), we consider the value of
A = 10 based the relative scale between the T.c and LD2; we find that the results relatively insensitive to X. We use 100 diffusion steps for DDIM and D2C unless mentioned otherwise, as running with longer steps is not computationally economical despite tiny gains in FID [84]. We include additional training details, such as architectures, optimizers and learning rates in Appendix C.
Table 2: Quality of representations and generations with LVGMs.
CIFAR-10 MSE 1
CIFAR- 100
fMoW
Model
FID 4
Acc t
FID 4
MSE +
Acc f
FID 4
MSE +
Acc f
NVAE [91]
36.4
0.25
18.8
42.5
0.53
4.1
82.25
0.30
27.7
DDIM [84]
4.16
2.5
22.5
10.16
3.2
2.2
37.74
3.0
23.5
D2 (Ours)
15.1
0.24
40.6
19.85
0.48
17.89
D2C (Ours)
10.15
0.76
76.02
14.62
0.44
42.75
44.7
2.33
66.9
7.1
Unconditional generation
For unconditional generation, measure the sample quality of images using the Frechet Inception Distance (FID, [40]) with 50,000 images. In particular, we extensively evaluate NVAE [91] and
Seee https::/ com /ttp///ab/////annnnnnnn issues for detailed description.

---

Figure 3: Generated samples on CIFAR-IO (left), fMoW (mid), and FFHQ 256 x 256 (right).
DDIM [84], competitive VAE model and a competitive diffusion model as baselines because we can directly obtain features from them without additional optimization steps". For them, we report mean-squared reconstruction error (MSE, summed over all pixels, pixels normalized to [0, 1J and linear classification accuracy (Acc., measured in percentage) over 2 features for the test set.
We report sample quality results' in Tables 2, and 3. For FID, we outperform NVAE in all datasets and outperform DDIM on CelebA-64 and SelebA-HQ-2566 which suggests our results are competitive with state-of of -ote--of-thhrrrrrreee
voersaralaliial generative models. In Table 2, we additionally compare NVAE, DDIM and D2C in terms of reconstruction and linear classification accuracy. As all three methods contain reconstruction losses, the MSE values are low and comparable. However, D2C enjoys much better linear classification accuracy than other two thanks the contrastive SSL component. We further note that training the same contrastive SSL method without Lnz achieves slightly higher 78.3% accuracy on CIFAR-10. We tried improving this via ResNet [38] encoders, but this significantly increased reconstruction error, possibly due to loss of information in average pooling layers.
Table 3: FID scores over different faces dataset with LVGMs.
Model
CelebA-64
CelebA-HQ-255.
FFHQ-256
NVAB [91] 13.48
40.26
26.02
DDIM [84]
6.53
25.6
D2C (Ours)
5.7
18.74
13.04
7.2 Few-shot conditional generation from examples
We demonstrate the advantage of D2C representations by performing few-shot conditional generation over labels. We consider two types of labeled examples: one has binary labels for which we train binary classifier; the other is positive-only labeled (e.g. images of female class) for which we train a PU classifier. Our goal here is to generate a diverse group of images with a certain label. We evaluate and compare three models: D2C. NVAE and DDIM. We train classifier 7 (c/ (z) over the latent space these models; we also train a image space classifier and use with DDIM (denoted as DDIM-I). We run Algorithm for these models, where line 4 is implemented via rejection sampling. As our goal is to compare different models, we leave more sophisticated methods [27] as future work.
First, we consider performing 8 conditional generation tasks over CelebA-64 with binary classifiers (trained over 100 samples, 50 for each class) and 4 PU classifiers (trained over 100 positively labeled and 10k unlabeled samples). We also report "naive" approach where we use all the training images (regardless of labels) and compute its FID with the corresponding subset of images (e.g., all images versus blond images). In Table 4, report the FID score between generated images (5k samples) and real images of the corresponding label.
Next, we perform a similar experiment CIFAR-10, 10, but with 50 labels for each of the 10 classes. For each label, we evaluate the FID between the real images and generated images under that label
For DDIM, the latent representations x (0) are obtained by reversing the neural ODE process.
"Due space limits, we place additional CIFAR- 10 results in Appendix D.

---

and take the average FID over the 10 labels. The average FID for D2C, DDIM, and NVAE are 32.57, 82.78, and 95.45 respectively (note that FIDs are expected be higher because evaluated with 5k real samples per class instead of the commonly used 50k). This shows that D2C performs significantly better than DDIM and NVAE on the few-shot conditional generation task for both datasets, illustrating the advantage of contrastive representations for few-shot conditional generation. We demonstrate some figures in the appendix.
Table 4: FID scores for few-shot conditional generation with various types of labeled examples. Naive performs very well for non-blond due to class percentages.
Method
Classes (% train set)
D2C
DDIM
NVAE
DDIM-I
Naive
Male (42%)
13.44
38.38
41.07
29.03
26.34
Female (58%)
9.51
19.25
16.57
15.17
18.72
Binary
Blond (15%)
17.61
31.39
31.24
29.09
27.51
Non-Blond (85%)
8.94
9.67
16.73
19.76
3.77
Male (42%)
16.39
37.03
42.78
19.60
26.34
Female (58%)
12.21
15.42
18.36
14.96
18.72
PU
Blond (15%)
10.09
30,20
31.06
76.52
27.51
Non-Blond (85%)
9.09
9.70
17.98
9.90
3.77
Fair generative modeling with D2C We illustrate some conditional generation results with D2C (from 100 "blond" labels) in Figure 4 (left), which exhibits biased gender ratio (3 examples out of 36). This kind of dataset bias in CelebA is well-documented in previous work [46]; among all the images with the "blond" attribute, only 5.6 are labeled as "male". In fact, we observe that the D2C samples have a similar gender bias as the original dataset since D2C has learned the data distribution quite well. To mitigate the presence of dataset bias in D2C's samples, we leverage a conditional generation approach. This is in contrast to previous methods that try to learn fair generative model via carefully designed representative dataset [20]. We give concrete instantiation of our method in the case of gender bias that can be generalized to all attributes. Suppose our goal is to achieve balanced gender ratio among all the generated images with "blond" attribute;
do by generating the same number samples conditioned on the (1) "blond, male" attributes and the (2) "blond, female" attributes. Figure 4 (right) demonstrates that D2C can be used for fair generative modeling by simply specifying our desired constraints with a handful of labels.
Blond
Blond Female
Blond Male
Figure 4: (Left) While D2C generations are faithful to the CelebA distributions, it also learns from the biases within the dataset. (Right) Conditional generation can make D2C a fairer generative model.
7.3 Few-shot conditional generation from manipulation constraints
Finally, we consider image manipulation where we use binary classifiers that are learned over 50 labeled instances for each class. To manipulate an image, we first acquire its latent code, move its latent code towards the decision boundary of the classifier and then use the diffusion prior to make the

---

Original
D2C
StyleGAN2
NVAE
DDIM
NN
Figure 5: Image manipulation results for blond (top) and red lipstick (bottom). D2C better than StyleGAN2 preserving details of the original image, such eyes, earrings, and background.
Red lipstick
Blond
DDIM
DDIM
NVAE
NVAE
StyleGAN2
StyleGAN2
0.0
0.2
0.3
0.4
0.5
0.0
0.1
0.3
Figure 6: AMT evaluation image manipulations. ær-axis shows the percentage that the evaluator selects the image generated from the corresponding model out of images from each model.
attributes in the CelebA-256 dataset, blond and red lipstick, over D2C, DDIM, NVAE and StyleGAN2 [50] (see Figure 5). The evaluation is bouble-blinded: neither nor the evaluators know the correspondence between generated image and underlying model during the study. We include more details (algorithm, setup and human evaluation) Appendix C and additional qualitative results (such beard and gender attributes) in Appendix D.
sample realistic. We perform Amazon Mechanical Turk (AMT) evaluations
In Figure 8, show the percentage of manipulations preferred by AMT evaluators for each model; D2C slightly outperforms StyleGAN2 for blond and significantly outperforms StyleGAN2 for red lipstick. When compare D2C with only StyleGAN2, D2C is preferred over 51.5% for blond and 60.8% for red lipstick. An additional advantage of D2C is that the manipulation is much faster than StyleGAN2, since the latter requires additional optimization reconstruction [106]. On the same Nvidia 1080Ti GPU, it takes 0,013 seconds to obtain the latent code D2C, while the same takes 8 seconds [106] for StyleGAN2 (615x slower). As decoding is very fast for both models, D2C generations are around two orders of magnitude faster to produce.
the latent space improve
Discussions and Limitations
We introduced D2C, VAE-based generative model with latent space suitable for few-shot condi- tional generation. To our best knowledge, our model is the first unconditional VAE demonstrate superior image manipulation performance than StyleGAN2, which is surprising given our use of regular NVAE architecture. We believe that with better architectures, such as designs from Style- GAN2 or Transformers [44], D2C can achieve even better performance. It is also interesting to formally investigate the integration berween D2C and other types of conditions on the latent space, as well as training D2C in conjunction with other domains and data modalities, such as text [74], in fashion that similar to semi-superee supervisd learning. Nevertheless, we note that our model have be used properly in order to mitigate potential negative socictal impacts, such deep fakes.
10

---

Acknowledgements
We thank the anonymous reviewers and Kristy Choi for insightful discussions and feedback.
Funding transparency statement. This research was supported by NSF (#1651565, #1522054, #1733686), ONR (N00014-19-1-11459, AFOSR (FA9550- 19-1002499 ARO (W91 Sloan Fellowship, Amazon AWS, Stanford Institute for memaan-Cetteredd Artificial Intelligence (HAI), and Google Cloud.
References
[1] Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing a broken ELBO, arXiv preprint arXiv:1711.0046: November 2017.
[2] Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. Sequential latent spaces for modeling the intention during diverse image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4261-4270, 2019.
[3] Jyoti Aneja, Alexander Schwing, Jan Kautz, and Arash Vahdat. NCP-VAE: Variational
autoencoders with noise contrastive priors. arXiv preprint arXiv:2010002911, October 2020.
[4] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv preprint arXiv:/701.078: January 2017.
[5] Muhammad Asim, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. arXiv preprint 19Xiv: 1672, May 2019.
[6] Sergey Bartunov and Dmitry Vetrov. Few-shot generative modelling with generative matching networks. In Amos Storkey and Fernando Perez-Cruz, editors, Proceedings of the Twenty- Twesty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 670-67 678. PMLR, 2018.
[7] David Bau, Jun- Zhu, Jonas Wulff, William Peebles, Hendrik Strobelt, Bolei Zhou, and Antonio Torralba. Seeing what gan cannot generate. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4502 4, 2019.
[8] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv: 1096, September 2018.
[9] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.0709. 2016.
[10] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are Few-Shot learners. arXiv preprint arXiv: 14165, May 2020.
[1]] Emmanuel J Candes, Justin K Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 59(8): (0(8))1207.. 1203 2006.
[12] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017.
[13] Ricky Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint urXiv:1806.07366 June 2018.

---

[14] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, February 2020.
[15] Xi Chen, Diederik Kingma, Tim Salimans, Yan Duan, Prafulla Dhariwal, John Schulman, llya Sutskever, and Pieter Abbeel. Variational lossy autoencoder. arXiv preprint arXiv:6611.02311, November 2016.
[16] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, March 2020.
[17] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 November 2020.
[18] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study training Self-Supervised vision transformers. arXiv preprini
April 2021.
[19] Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:201 1. 2020.
[20] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon. Fair generative modeling via weak supervision. In International Conference on Machine Learning, pages 1887-1898. PMLR, 2020.
[21] Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6172-61 800, 2018.
[22] Louis Clouâtre and Marc Demers. Figr: Few-shot image generation with reptile. arXiv preprint arXiv:1901.021999 2019.
[23] Bin Dai and David Wipf. Diagnosing and enhancing VAE models. arXiv preprint
arXiv:1903.05783 March 2019.
[24] Giannis Daras, Joseph Dean, Ajil Jalal, and Alexandros G Dimakis. Intermediate layer opti- mization for inverse problems using deep generative models. arXiv preprint arXiv:2102.07555 02440364 February 2021.
[25] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 140.00000 October 2018
[26] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint WwXiv:205500034: 2020.
[27] Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint 1xXi:22055002233 May 2021.
[28] L Dinh, Krueger, and Bengio. NICE: Non-linear independent components estimation. arXiv preprint arXiv: 2014.
[29] Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio. Density estimation using real NVP. arXiv preprint 1605v: May 2016.
[30] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint arXiv: 1605.00979 May 2016.
[31] Yilun Du and Igor Mordatch. Implicit generation and generalization in Energy-Bass -Based models. arXiv preprint wrXiv:1903.08683 March 2019.
[32] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Mar- tin Arjovsky, and Aaron Courville. Adversarially learned inference. arXiv preprint arXiv:1606 00704, June 2016.
12

---

[33] Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In 14th ACM SIGKDD, pages 213-220, 2008.
[34] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M Welling, C Cortes, N D Lawrence, and K Q Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672-2680. Curran Associates, Inc., 2014.
[35] Jean-Baste Mann-Bstteen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to Self-Supervised learning. arXiv preprint arXiv:200..07733, June 2020.
[36] Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International conference on machine learning, pages 2444-2444. PMLR, 2019.
[37] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, November 2019.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03388) December 2015.
[39] Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In PMLR, 2020.
International Conference on Machine Learning, pages
[40] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two Time-Scale update rule converge to a local nash equilibrium. arXiv preprint 1056::7770.85000000 June 2017.
[41] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1771-107,) August 2002.
[42] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, arXiv preprint arXiv:2006. /239, June 2020.
[43] Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS, volume 1, page 2. approximateinferrneee.oo 2016.
[44] Drew A Hudson and C Lawrence Zitmick. Generative adversarial transformers. arXiv preprint 103.v:2103.012000 2021.
[45] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
[46] Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. Selective classification can magnify disparities across groups. arXiv preprint arXiv:2010. 14134, 2020.
[47] Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640, July 2020.
[48] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 October 2017.
[49] Tero Karras, Samuli Laine, and Timo Aila. A Style- Style-Based generator architecture for generative adversarial networks. arXiv preprint / 8.40449 December 2018
[50] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 8110-8119, 2020.
13

---

[51] Diederik P Kingma, Tim Salimans, Rafal Jozefowicz. Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In D Lee, M Sugiyama, U V Luxburg, Guyon, and R Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4743 Curran Associates, Inc., 2016.
[52] Diederik P Kingma and Max Welling. Auto-Encoding variational bayes. arXiv preprint arXiv:1312.6114vv December 2013.
[53] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convolutional neural networks. In F Pereira, C C Burges. L Bottou, and K Q Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097-1105. Curran Associates, Inc., 2012.
[54] Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. In International conference machine learning, pages 1558-156.. PMLR, 2016.
[55] Li, H Liu, C Chen, Y Pu, L Chen, and others. Alice: Towards understanding adversarial learning for joint distribution matching. Advances in neural information processing systems, 2017.
[56) Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Optimus: Organizing sentences via pre-trained modeling of a latent space. arXiv preprint 0rXiv:2004.04000 2020.
[57] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[58] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738. 2015.
[59] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1511.05644 2015.
[60] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Generating images from captions with attention. arXiv preprint 1XXiv:5511.0299. 2015.
[61] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1788 November 2014.
[62] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprint arXiv:2103.16091, March 2021.
[63] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normaliza- tion for generative adversarial networks. arXiv preprint arXiv:1802.05957, February 2018.
[64] Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv: October 2016.
[65] Alex Najibi. Racial Discrimination Face Recognition Technology, 2020.
[66] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4467-4477, 2017.
[67] Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling. International Conference on Artificial Intelligence and Statistics, pages 4474 4484. PMLR, 2020.
[68] Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In European conference on computer vision, pages 69-84. Springer, 2016.
14

---

[69] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A Efros, and Richard Zhang. Swapping autoencoder for deep image manipulation. arXiv preprint urXiv:2007.00653 July 2020.
[70] Or Patashmik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text-Driven manipulation of StyleGAN imagery. arXiv preprint arXiv:2103.17249 March 2021.
[71] Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. Waveflow: A compact flow-based model for raw audio. In International Conference on Machine Learning, pages 7706-7716. PMLR, 2020.
[72] Tiziano Portenier, Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker. Faceshop: Deep sketch-based face image editing. ACM Transactions on Graphics, 37(4), 2018.
[73] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 0000000000, February 2021.
[74] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-Shot Text-to-lmage generation. arXiv preprint arXiv:2 12092, February 2021.
[75] Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. arXiv preprint arXiv: 906.00446, 2019.
[76] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pages 1060-1069. PMLR, 2016.
[77] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows arXiv preprint arXiv:1505.05770, May 2015.
[78] Gareth O Roberts and Jeffrey S Rosenthal. Optimal scaling of discrete approximations to langevin diffusions. Journal of the Royal Statistical Society: Series (Statistical Methodology). 60(1):255-268, 1998.
[79] Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution matching in variational inference. arXiv preprint 1aM.:::802.6444 February 2018
[80] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243-9252, 2020.
[81] Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. arXiv preprint March 2015.
[82] Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. arXiv preprint D'Xiv:2007009852 2020.
[83] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. arXiv preprint
December 2018.
[84] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[85] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint urXiv:1907.056000 July 2019.
[86] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
15

---

[87] Hiroshi Takahashi, Tomoharu Iwata, Yuki Yamanaka, Masanori Yamada, and Satoshi Yagi. Variational autoencoder with implicit optimal priors. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01) (:00)):066-5000) 5073, July 2019.
[88] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein Auto-Encoders. arXiv preprint arXiv: November 2017.
[89] Jakub M Tomezak and Max Welling- Improving variational auto-encoders using householder flow. arXiv preprint orXhv:1611.09600 2016.
[90] Jakub M Tomezak and Max Welling. VAE with a VampPrior. arXiv preprint teXiiv/705.07) May 2017.
[91] Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. arXiv preprint 20:::0007703898, July 2020.
[92] Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. arXiv preprint arXiv:2106.00 05931, 2021.
[93] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint 1609.
September 2016.
[94] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. arXiv preprint arXiv: 1601.06759 January 2016.
[95] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807.0000000000 000007,,.7,, July 2018.
[96] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint (tfi:1711.00900 November 2017.
[97] Ting-Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
[98] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. TediGAN: Text-Guided diverse face image generation and manipulation. 2021.
[99] Zhe Xie, Chengxuan Liu, Yichi Zhang, Hongtao Lu, Dong Wang, and Yue Ding. Adversar- ial and contrastive variational autoencoder for sequential recommendation. arXiv preprint arXiv:2103 10693, March 2021.
[100] Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. Generative hierarchical features from synthesizing images. arXiv e-prints, pages arXiv-2007, 2020.
[101] Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph convolutional pol- icy network for Goal-Directed molecular graph generation. arXiv preprint 10X.:2300..02473 June 2018
[102] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Proceedings of the AAAI conference on artificial intelligence. volume 31, 2017.
[103) Shengjia Zhao, Jiaming Song, and Stefano Ermon. InfoVAE: Information maximizing varia- tional autoencoders. arXiv preprint
June 2017.
[104] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Towards deeper understanding of variational autoencoding models. arXiv preprini arXiv: February 2017.
[105] Shengjia Zhao, Jiaming Song, and Stefano Ermon. A lagrangian perspective on latent variable generative models. In Proc. 34th Conference on Uncertainty in Artificial Intelligence, 2018.
[106] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European Conference on Computer Vision, pages 992-608. Springer, 2020.
[107] Jiapeng Zhu, Deli Zhao, Bolei Zhou, and Bo Zhang. Lia: Latently invertible autoencoder with adversarial learning. 2019.
16


Reviews:
[{"review_id": "voydCe4HpWp", "reviewer": "Reviewer_n2do", "summary": "This paper proposes to learn a joint model consisting of a VAE and a latent space diffusion. The training process is augmented with a contrastive loss.\n\nThe authors show that this approach makes possible a few-shot generation of images with desired properties. To achieve this,  one needs to train a classifier in the latent space and after that sample latent codes with high classifier scores in any way.\n\nAlso, the case of semantic editing is evaluated. The manipulation is implemented using a Langevin-like approach in the latent space.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Strengths.\n\n    To the best of my knowledge, the proposed joint training of VAE and latent diffusion model is relatively novel, though the motivation for such an approach is pretty clear. Application of the contrastive loss to the diffusion model also has not been described earlier.  Also, I like the way the authors handle the problem of latent posterior mismatch, Theorem 2 is of particular interest. \n\n2. Questions.\n\n    a. First of all, I wonder which properties of diffusion make it a better candidate for latent posterior mismatch than, e.g. a normalizing flow? While training a diffusion model still contains a lot of heuristics (e.g. the usage of a so-called simple objective instead of a theoretically motivated variational bound is still a common choice [1] on par with synthetic hybrid objectives [2]), the training process of NF seems more stable. By the way, what weighting functions $w (\\alpha\\_i)$ (equation 2) were used during the training of D2C?\n\n    b. Though the authors claim that the usage of contrastive loss was beneficial, (line 250, 269), there is no ablation study that proves that.\n\n    c. Examples of image manipulation, presented in the paper, are not so impressive. To my mind, an example of gender editing, shown in the Appendix, is the most interesting one, while others are too simplistic. I think it could be more interesting to evaluate the presented approach on the 'smile' attribute or other attributes that require more complicated transformation that at the same time preserves the human identity.\n\n    d. Figure 13b of the Appendix contains just a single image of a blond man. Is it due to the bias of labeled data? Or does this disproportion have another explanation?\n\n3. Typos\n    a. Equation 5: p -> p_\\theta\n\n    b. Line 138: x_0 -> z_0\n\n\n4. Rating (pre-rebuttal)\n\n    To my mind, this paper has demonstrated appealing results, but a more thorough evaluation of the proposed model's properties is required.\n\n5. Updates (post-rebuttal)\n\n   After reading other reviews and the authors' feedback, I retain my original assessment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No suggestions", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "oqFvtk6IdA6", "reviewer": "Reviewer_5hPh", "summary": "The paper describes two methods: 1) how to incorporate self-supervised learning in VAE to have a meaningful encoded representation, 2) how to use diffusion model over later to bridge the gap between posterior and prior for unconditional generation, 3) how to use the given setup to do few-shot conditional generation. Empirically, authors show highly improved representation quality, which can be attributed to the incorporation of self-supervised learning. Additionally, experiments show improved FID scores for face datasets, but not for CIFAR. Authors don’t show results on ImageNet. The authors show improved FID scores for the conditional generation task. There is no FID comparison against StyleGAN2 (or any other GAN-based model), even if the authors claim to come very close in sample quality to such models. The qualitative depiction of image manipulation tasks is very impressive.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is a delight to read. All components are explained well, and the arguments are easy to follow. The contributions are novel and will be valuable to the rest of the community.\n\nPros:\n1. The proposed approach seems to alleviate the prior hole problem for VAEs. However, it comes at the cost of deviating from optimizing the true lower bound on likelihood. FID comparison against GAN models will strengthen the paper.\n2. The proposed approach is much faster at image manipulation tasks, which could be very valuable for many applications.\n\nCons:\n1. Addition of self-supervised learning and diffusion model over latent is confounding. It would be great to study the effectiveness of diffusion model over latent as a solution to the prior hole problem in VAEs.\n2. FID score comparison against GANs for image quality is warranted since authors claim to come close to GANs in sample quality.\n3. Discussion about the strength/weakness of the model with varying data diversity would be useful since the FID scores don’t improve for non-face datasets.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address the limitations and societal impact sufficiently.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "m5T90rb-Z7j", "reviewer": "Reviewer_Sq7M", "summary": "The authors present Diffusion Decoding Models with Contrastive Representations (D2C) - a special VAE model with diffusion modeling in the latent space combined with contrastive learning based training of the inference network. The authors claim that D2C helps avoid the prior hole issue with VAEs due to diffusion modeling of the latent space, while contrastive learning helps learn meaningful representations allowing Few-Shot conditional generation. They further demonstrate that D2C outperforms NVAE and DDIM in few-shot conditional generation while remaining comparable to DDIM in unconditional generation. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper applies 1) existing methods of diffusion models in the latent space of a VAE model, and 2) contrastive representation learning methods in the latent space of VAE. \n- All sections of the paper are clearly written and easy to follow. \n- D2C achieves strong results on few-shot conditional generation compared to a strong VAE baseline. \n- In image manipulation experiments, while StyleGAN2 generates more \"realistic looking images\", they are poor at preserving details. D2C ,on the other hand, seems to preserve features but may produce distorted images (Figure 1 bottom image). So, there is a tradeoff between these two methods. However, D2C does perform qualitatively better manipulations than NVAE. \n- It would be great to perform some analysis on the role of diffusion modeling in the latent space vs contrastive representation learning methods aka the two main modifications over a standard VAE. E.g. how good is a NVAE model when trained with an auxiliary contrastive representation learning objective? Such an analysis would make the contributions more significant for future works in terms of recognizing the importance of each component in latent variable generative models. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately address the limitations and potential negative societal impact of their work.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Discrimination / Bias / Fairness Concerns"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SseVwiHfL1K", "reviewer": "Reviewer_28ig", "summary": "D2C proposes three main contributions to diffusion models, a recently popularized class of likelihood-based generative models based on parameterized Markov chains, particularly successful when viewed from a denoising score matching lens. First, this paper learns a prior for the *latent space* of an image VAE (NVAE), while past work mostly applied diffusion to the data space. Second, D2C adds a SimCLR-style contrastive loss to image latents in order to improve the quality of latent representations for classification. Third, binary attribute classifiers are learned conditioned on these latents and used to guide prior sampling with rejection sampling or Langevin dynamics. Experimental results qualitatively and quantitatively show some success in conditional image generation and manipulation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper with novel contributions in its hybridization of single modality contrastive representation learning, image generation and conditional guidance. Its findings on the success of diffusion in latent space should be exciting to researchers in the field. The writing is clear. In my view, it makes good contributions, but can be improved significantly in experimental methodology, breadth of experiments, and fair comparisons.\n\nReference [59], on Symbolic Music Generation, is more relevant than is given credit in the paper. [59] also develops a diffusion prior for VAEs, and this should be made clear.\n\nIs the MoCo contrastive loss computed with augmentations to images or directly to latents? Does this apply during NVAE pre-training?\n\nIn 7.1, is it possible to measure the reconstruction error of D2C without L_D2? An ablation of L_C in terms of FID, MSE and Accuracy would also add quite a bit of insight, as the addition of the contrastive loss is a core contribution of the work. Is D2C FID’s worse than DDIM on CIFAR and fMoW due to the contrastive loss, the proposed diffusion in latent space, or other architectural details? Currently, these contributions are entangled, and their relative contributions are unclear to me. Clarifying this is one of my main concerns.\n\nMy other main concern is the limited breadth of conditional generation and manipulation tasks. Only four attributes are considered in the paper -- male/female, beard/no beard, lipstick/no lipstick, and blond/non-blond, all on face datasets. Does the method improve class conditional generation e.g. for CIFAR-10, or manipulations on fMoW?\n\nAlso, how do GAN models compare on attribute conditional image generation in this few-shot setting?\n\nAdditional unconditional generation baselines should be added to Table 3, such as StyleGAN2. These numbers should be able to be copied from their respective papers, and would help contextualize the results. It’s understandable that encoder-free approaches are excluded from Table 2 due to the reconstruction MSE metric, but even there, 76% accuracy on CIFAR-10 is far from the state of the art (e.g. Image GPT has higher than 90% without any contrastive loss).\n\n# Minor comments\nWhy do conditional generations use rejection sampling, while image manipulations use langevin sampling from the reweighted density? It seems rejection sampling would become inefficient with higher class counts than the binary attribute settings considered in the paper.\n\nThe notation flips the usual time convention for diffusion models, where during generation time runs backward from T to 0. In your exposition, the alpha parameter is varied from 0 to 1 during latent generation. It’s OK since you define the convention, but is a bit confusing.\n\nAlg 1 L2 should have q_\\phi(z^1|x_i) not x", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously believe that the paper should be accepted. The paper focuses on an important problem and some interesting ideas such as guided diffusion and the use of self-supervised representations has been proposed. There are complaints about the quality of the baselines and the rigorousness of the evaluation. The highlighted results do show some artifacts, and the proposed approach sounds like a combination of several disjoint technique. Hence, I recommend acceptance as a poster.
Decision: Accept (Poster)