Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes two methods: 1) how to incorporate self-supervised learning in VAE to have a meaningful encoded representation, 2) how to use diffusion model over later to bridge the gap between posterior and prior for unconditional generation, 3) how to use the given setup to do few-shot conditional generation. Empirically, authors show highly improved representation quality, which can be attributed to the incorporation of self-supervised learning. Additionally, experiments show improved FID scores for face datasets, but not for CIFAR. Authors don’t show results on ImageNet. The authors show improved FID scores for the conditional generation task. There is no FID comparison against StyleGAN2 (or any other GAN-based model), even if the authors claim to come very close in sample quality to such models. The qualitative depiction of image manipulation tasks is very impressive.
main_review: 
The paper is a delight to read. All components are explained well, and the arguments are easy to follow. The contributions are novel and will be valuable to the rest of the community.

Pros:
1. The proposed approach seems to alleviate the prior hole problem for VAEs. However, it comes at the cost of deviating from optimizing the true lower bound on likelihood. FID comparison against GAN models will strengthen the paper.
2. The proposed approach is much faster at image manipulation tasks, which could be very valuable for many applications.

Cons:
1. Addition of self-supervised learning and diffusion model over latent is confounding. It would be great to study the effectiveness of diffusion model over latent as a solution to the prior hole problem in VAEs.
2. FID score comparison against GANs for image quality is warranted since authors claim to come close to GANs in sample quality.
3. Discussion about the strength/weakness of the model with varying data diversity would be useful since the FID scores don’t improve for non-face datasets.
limitations_and_societal_impact: The authors address the limitations and societal impact sufficiently.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present Diffusion Decoding Models with Contrastive Representations (D2C) - a special VAE model with diffusion modeling in the latent space combined with contrastive learning based training of the inference network. The authors claim that D2C helps avoid the prior hole issue with VAEs due to diffusion modeling of the latent space, while contrastive learning helps learn meaningful representations allowing Few-Shot conditional generation. They further demonstrate that D2C outperforms NVAE and DDIM in few-shot conditional generation while remaining comparable to DDIM in unconditional generation. 
main_review: - The paper applies 1) existing methods of diffusion models in the latent space of a VAE model, and 2) contrastive representation learning methods in the latent space of VAE. 
- All sections of the paper are clearly written and easy to follow. 
- D2C achieves strong results on few-shot conditional generation compared to a strong VAE baseline. 
- In image manipulation experiments, while StyleGAN2 generates more "realistic looking images", they are poor at preserving details. D2C ,on the other hand, seems to preserve features but may produce distorted images (Figure 1 bottom image). So, there is a tradeoff between these two methods. However, D2C does perform qualitatively better manipulations than NVAE. 
- It would be great to perform some analysis on the role of diffusion modeling in the latent space vs contrastive representation learning methods aka the two main modifications over a standard VAE. E.g. how good is a NVAE model when trained with an auxiliary contrastive representation learning objective? Such an analysis would make the contributions more significant for future works in terms of recognizing the importance of each component in latent variable generative models. 
limitations_and_societal_impact: The authors adequately address the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: D2C proposes three main contributions to diffusion models, a recently popularized class of likelihood-based generative models based on parameterized Markov chains, particularly successful when viewed from a denoising score matching lens. First, this paper learns a prior for the *latent space* of an image VAE (NVAE), while past work mostly applied diffusion to the data space. Second, D2C adds a SimCLR-style contrastive loss to image latents in order to improve the quality of latent representations for classification. Third, binary attribute classifiers are learned conditioned on these latents and used to guide prior sampling with rejection sampling or Langevin dynamics. Experimental results qualitatively and quantitatively show some success in conditional image generation and manipulation.
main_review: This is an interesting paper with novel contributions in its hybridization of single modality contrastive representation learning, image generation and conditional guidance. Its findings on the success of diffusion in latent space should be exciting to researchers in the field. The writing is clear. In my view, it makes good contributions, but can be improved significantly in experimental methodology, breadth of experiments, and fair comparisons.

Reference [59], on Symbolic Music Generation, is more relevant than is given credit in the paper. [59] also develops a diffusion prior for VAEs, and this should be made clear.

Is the MoCo contrastive loss computed with augmentations to images or directly to latents? Does this apply during NVAE pre-training?

In 7.1, is it possible to measure the reconstruction error of D2C without L_D2? An ablation of L_C in terms of FID, MSE and Accuracy would also add quite a bit of insight, as the addition of the contrastive loss is a core contribution of the work. Is D2C FID’s worse than DDIM on CIFAR and fMoW due to the contrastive loss, the proposed diffusion in latent space, or other architectural details? Currently, these contributions are entangled, and their relative contributions are unclear to me. Clarifying this is one of my main concerns.

My other main concern is the limited breadth of conditional generation and manipulation tasks. Only four attributes are considered in the paper -- male/female, beard/no beard, lipstick/no lipstick, and blond/non-blond, all on face datasets. Does the method improve class conditional generation e.g. for CIFAR-10, or manipulations on fMoW?

Also, how do GAN models compare on attribute conditional image generation in this few-shot setting?

Additional unconditional generation baselines should be added to Table 3, such as StyleGAN2. These numbers should be able to be copied from their respective papers, and would help contextualize the results. It’s understandable that encoder-free approaches are excluded from Table 2 due to the reconstruction MSE metric, but even there, 76% accuracy on CIFAR-10 is far from the state of the art (e.g. Image GPT has higher than 90% without any contrastive loss).

# Minor comments
Why do conditional generations use rejection sampling, while image manipulations use langevin sampling from the reweighted density? It seems rejection sampling would become inefficient with higher class counts than the binary attribute settings considered in the paper.

The notation flips the usual time convention for diffusion models, where during generation time runs backward from T to 0. In your exposition, the alpha parameter is varied from 0 to 1 during latent generation. It’s OK since you define the convention, but is a bit confusing.

Alg 1 L2 should have q_\phi(z^1|x_i) not x
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to learn a joint model consisting of a VAE and a latent space diffusion. The training process is augmented with a contrastive loss.

The authors show that this approach makes possible a few-shot generation of images with desired properties. To achieve this,  one needs to train a classifier in the latent space and after that sample latent codes with high classifier scores in any way.

Also, the case of semantic editing is evaluated. The manipulation is implemented using a Langevin-like approach in the latent space.
main_review: 1. Strengths.

    To the best of my knowledge, the proposed joint training of VAE and latent diffusion model is relatively novel, though the motivation for such an approach is pretty clear. Application of the contrastive loss to the diffusion model also has not been described earlier.  Also, I like the way the authors handle the problem of latent posterior mismatch, Theorem 2 is of particular interest. 

2. Questions.

    a. First of all, I wonder which properties of diffusion make it a better candidate for latent posterior mismatch than, e.g. a normalizing flow? While training a diffusion model still contains a lot of heuristics (e.g. the usage of a so-called simple objective instead of a theoretically motivated variational bound is still a common choice [1] on par with synthetic hybrid objectives [2]), the training process of NF seems more stable. By the way, what weighting functions $w (\alpha\_i)$ (equation 2) were used during the training of D2C?

    b. Though the authors claim that the usage of contrastive loss was beneficial, (line 250, 269), there is no ablation study that proves that.

    c. Examples of image manipulation, presented in the paper, are not so impressive. To my mind, an example of gender editing, shown in the Appendix, is the most interesting one, while others are too simplistic. I think it could be more interesting to evaluate the presented approach on the 'smile' attribute or other attributes that require more complicated transformation that at the same time preserves the human identity.

    d. Figure 13b of the Appendix contains just a single image of a blond man. Is it due to the bias of labeled data? Or does this disproportion have another explanation?

3. Typos
    a. Equation 5: p -> p_\theta

    b. Line 138: x_0 -> z_0


4. Rating (pre-rebuttal)

    To my mind, this paper has demonstrated appealing results, but a more thorough evaluation of the proposed model's properties is required.

5. Updates (post-rebuttal)

   After reading other reviews and the authors' feedback, I retain my original assessment.
limitations_and_societal_impact: No suggestions
review_title: null

Meta-review: The reviewers unanimously believe that the paper should be accepted. The paper focuses on an important problem and some interesting ideas such as guided diffusion and the use of self-supervised representations has been proposed. There are complaints about the quality of the baselines and the rigorousness of the evaluation. The highlighted results do show some artifacts, and the proposed approach sounds like a combination of several disjoint technique. Hence, I recommend acceptance as a poster.
Decision: Accept (Poster)