Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a generative model based on the VAE framework with a sophisticated prior / inference scheme based on denoising score-matching. Authors develop extensive machinery to adapt related ideas from score-based generation in the observed space to latent space. This greatly improves sampling time while preserving and sometimes improving sample quality and/or likelihood of data. I believe the paper is an interesting addition to the existing collection of generative modelling techniques.
main_review: Novelty:

To my knowledge, the method is novel. There is a certain intersection with some concurrent NeurIPS submissions though which will can be reflected by authors when preparing the final version of the paper.

Clarity:

The paper is generally well-written but due a large amount of mathematical material, the writing is dense and there are some details that I would to be discussed more. 

1) As I understand, the exceptionally good results with modelling binary images are explained by the fact that score-matching in the latent space resulted into very tight variational approximation and, hence, good gradients for training the decoder. Would it be possible to quantify this directly by measuring the KL divergence for the LSGM and the same model trained with standard amortized inference techniques? Or there are even better ways to further illustrate this?
2) I couldn't find detailed information on how the structure of the latent space. As I understand, it's a lower resolution image-like structure. If so, can authors visualize evolution of the latent variable under the SDE and what reconstructions does it produce? 
3) Clearly, effectiveness of LSGMs depends on the choice of the latent space, and I would like to get, again, a more detailed discussion around this. How do the performance metrics depend on its size and structure (image-like vs flat vector)?

Quality:

I didn't read proofs for the presented theorems, but I followed equations in the main text and they made sense to me.

Significance:

I believe LSGMs are a valuable contribution because it can enable faster generation and tighter variational approximations.

I think the potential impact can be further improved if similarly better results are obtained with decoder architectures othen than NVAE which is arguably very specially structured in handling latent variables.
limitations_and_societal_impact: No issues.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to learn a flexible VAE prior using score-based generative model. Contrary to the existing score-based model that builds on high-dim pixel space, the paper applies the score-based model directly on the latent space which is typically low-dimensional.  The effectiveness of the model has been verified using generative FID score, negative log-likelihood as well as various ablation studies. 
main_review: Originality: 

 (+/-) The model presented is relatively intuitive and seems not appeared in the recent literature. However, the idea that uses some recently well-performed diffusion/score-based model as a flexible prior of VAE is straightforward and appears less surprising. 

Quality: 

 (+) The authors put efforts on designing various ablation studies to explore their model and conduct quite a lot comparisons with existing baselines. The overall generative performance looks good. I also appreciate the authors provide comprehensive supplementary materials for additional proof, details and results. 

 (-) Though the FID score is good, but the vae and Sgm backbone structure can be complicated, and some places need better described in the main text. I understand the powerful backbone structures are used to target SOTA performance. But from the model/methodology side, it might be better to also focus on the "basic" backbone structures on which quite a few existing generative models build (e.g., the vae backbone with a few convolution-relu layers). Does the paper try such basic backbone models in the experiments? In the tab 2& 3, Ours (VAE backbone) refers to the non-NVAE structure? What structure exactly? Also, does all the results obtained by pre-training the NVAE first OR jointly train both VAE and SGM? Would be more interesting if the latter.  I would adjust my initial scores based on the feedback of the authors on above issues. 

Clarity: 
 
(+) overall, the paper is easy to follow and presentation is clear despite a few places about training procedures as listed above. 

Significance: 

 (+/-) The idea is intuitive and could be easy to use/adapt for the follow-up works, but the idea itself is not so surprising, and cannot be considered transformative.
 

limitations_and_societal_impact: The paper mentions its limitation and broader impact.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes Latent Score-based Generative Model (LSGM) which introduces a score-based prior in the Variational Autoencoder (VAE) framework. In contrast to previous score-based models that operate in the data space, LSGM uses a score-based model in the latent space. A sample from a base distribution (standard Gaussian) is denoised using the score-based prior in the latent space and is then mapped to the data space using a decoder. The authors further discuss how the various terms in the ELBO can be computed to train the model in an end-to-end fashion without requiring the time-dependent marginal score function. Multiple variance reduction techniques and training tricks have also been proposed for the resulting objective function. Both quantitative and qualitative results demonstrate the ability of the model to generate high fidelity images.
main_review: ### Significance

This paper makes a significant contribution both to the fields of score-based models and VAEs with more expressive priors. Severals well-motivated training stabilization techniques have been proposed which may prove helpful for the community. The paper also improves the state of the art in terms of sample quality (FID) for VAEs.

### Clarity

The paper is well-written and organized. There are a fews places where inclusion of details in the main text would improve the clarity of the paper:

i) Line 148: The parameterization of score function using $\epsilon_\theta$.    
ii) Line 263: Differences between the three models.

### Technical quality

The paper is technically sound. The empirical results are fairly comprehensive and evaluate LSGM on various datasets against state-of-the-art deep generative models. Ablation studies presented help elucidate the contribution of different components of the proposed model.

### Originality/Relation to prior work

The paper presents a novel combination of score-based modeling with VAEs: a score-based prior distribution is introduced in the VAE framework. The relation to prior-work has been discussed sufficiently. 

The proposed work is also related to recent works [e.g., 1, 2, 3] that propose techniques to improve samples from deep generative models (particularly GANs). In these works, latent vectors are first diffused to a "better" point and then decoded to generate a sample. This is similar to LSGM where the latent vectors are denoised using a score-based model. A discussion of relation to these works would improve the related works section.

[1] Tanaka, Akinori. "Discriminator optimal transport." arXiv preprint arXiv:1910.06832 (2019).

[2] Ansari, Abdul Fatir, Ming Liang Ang, and Harold Soh. "Refining deep generative models via discriminator gradient flow." arXiv preprint arXiv:2012.00780 (2020).

[3] Che, Tong, et al. "Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling." arXiv preprint arXiv:2003.06060 (2020).

### Additional comments/Questions

- Did the authors investigate what final value of $\alpha$ is learned by the model? Specifically, is there an "optimal" mixing coefficient?
- In the ablation study the authors mention that the mixture formulation is particularly important for the sample quality. Do the authors have any insights on why this is the case?

-------
Post Rebuttal: Thank you for the clarification on $\alpha$ and the mixture formulation. I believe that a detailed discussion of the mixture formulation will improve the manuscript. 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper at hand proposes end-to-end training of score-based generative models in the latent space of a variational autoencoder. The VAE is pre-trained using a normal prior which is then replaced by the score-based generative model that is then jointly trained with the VAE. The paper also introduces a novel training objective based on the cross entropy between the encoder distribution and the SGM prior. Two techniques for variance reduction of the loss function are discussed.

main_review: REASONS FOR SCORE:

- The idea of applying the score based model to the latent space of a VAE is novel, as far as I can see.
- The experimental evaluation yields promising results on 5-bit CIFAR10, CelebA and binarized MNIST and OMNIGLOT: the proposed LSGM outperforms a variety of generative models in terms of test likelihood and FID-score (although only by a very small margin). The value of the proposed model lies within the sampling time which is decreased from 44.6 min to to 3.91 minutes for a batch of 16 images, which is remarkable.
- The choice of models that are compared is, in my opinion, complete and the experimental evaluation is overall very convincing. Only [1] achieve slightly better results on CelebA, however the paper appeared on arxiv after the NeurIPS deadline. In Table 3 the authors report an FID score of 10.70 on CelebA for [2], however in Table 3 of [2] an FID of 10.2 is reported (again this last version was uploaded after the deadline). 
- The structure of the paper is reasonable. Although the paper is very technical, the authors succeeded at clearly explaining their approach. A short background section introduces the relevant concepts for readers that are not familiar with score-based generative models. During reading the paper, I spotted no typos.
- The authors justify their choices by an ablation study where they analyze the effect of SDEs, training objectives, weighting mechanisms, and variance reduction techniques. As a small point of criticism: I had a hard time understanding Table 6. The authors should spend some time refactoring the table and its very long caption.

CONCLUSION:

Overall, I would reccomend to accept this submission into NeurIPS 2021. The paper is well-written and addresses the efficiency issues of score-based generative models. The experimental results on the benchmark datasets are promissing.

MINOR REMARKS:

- Paragraph captions not always in title-case (for example: "Implementation details")


REFERENCES:

[1] Kim, Dongjun, et al. "Score Matching Model for Unbounded Data Score." arXiv preprint arXiv:2106.05527 (2021).

[2] Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." arXiv preprint arXiv:2012.09841 (2021)
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The reviewers generally agreed that this was a novel and exciting contribution. The author-reviewer discussion further supported the paper's quality.
Decision: Accept (Poster)