Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses posterior collapse in VAE and also tries to mitigate the issue with many existing solutions for this which is the trade off with poor fit to the prior.

The authors propose a novel regularization to substitute the KL regularization in ELBo for VAEs, which is based on the density gap between the aggregated posterior and the prior. Since quantities related to aggregated posterior need for the regularizer depend on the whole dataset are expensive to compute, this paper further changes the objective to consider aggregation over minibatches only. 

This regularizer maximizes the ELBO as well as the mutual information between the input and the latent variable.

For Gaussian settings, the authors present a regularizer for more aggressive mutual information between the latent variable and data by imposing a regularizer over marginal distributions over each dimension of the latent variable.

The empirical comparison is done with relevant baselines on text datasets.

strengths_and_weaknesses: The paper is sound and the baselines are well chosen covering a range of related work.

The technical contribution hinges very heavily on the paper by Hoffman et al.'s formulation, but I believe the proposed regularizer is novel.

The interpolation study shows is interesting and the proposed approach seems to be better than the baselines on the chosen metric for interpolation. However, this metric is not directly related to mutual information between latent variables and text or the diversity one can expect from the samples of the generative model.

My main concern is that the main empirical results show that baseline/competing models are as good/better than the proposed method -- especially BN-VAE. The only metric BN-VAE does slightly worse is CU, and I am unsure how to interpret this unconventional metric.

The proposed approach also is more expensive due to the quantities involved with the aggregated posterior. A comparison of runtime with respect to other approaches would be helpful.

Some related work:
VAE with a VampPrior, Tomczak et al.
Learning to Explain: An Information-Theoretic Perspective on Model Interpretation, Chen et al.
questions: some notes on the presentation:
Line 53: q(z|x) instead of y, rather x and y are confusing, its the same thing
Indicator operator used for MI
Line 82
Text in math is poorly formatted

limitations: Please address the runtime of the proposed approach.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on a well-known problem in variational auto-encoders: the learned latent representation often has "gaps" in its prior, i.e. there are latent samples with high prior PDF values that fail to generate coherent outputs.

The authors analyse the problem and propose a novel objective to fix the issue that combines the ELBO with a mutual information term. Importantly, the proposed surrogate objective is well-motivated.
strengths_and_weaknesses: **Strengths**

- important problem in deep generative modeling
- contribution is interesting and well-motivated
- good experimental section

**Weaknesses**

- not sure to understand whether the proposed objective is still a lower bound to the log-likelihood
- weird notation, e.g. the use of X and Y variables, it should be X only, I don't understand why to introduce a second RV for the same observation. Also, q(n) is the same as p(n), no? Why a proposal distribution here? It is fixed(?)

**Missing citations**

The following citations are missing and they provide a broader view on how researchers tackle the problem in the NLP community:
- SentenceMIM: A Latent Variable Language Model (Micha Livne, Kevin Swersky, David J. Fleet)
- A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text (Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, Yiming Yang)
- Preventing posterior collapse in variational autoencoders for text generation via decoder regularization (Alban Petit, Caio Corro)
- Preventing Posterior Collapse with Levenshtein Variational Autoencoder (Serhii Havrylov, Ivan Titov)
questions: - is the proposed objective still a lower bound to the log-likelihood?
- l. 123: I don't understand what is the point of stating that "we only consider  z \in ..." => q(z) is Gaussian, so we can't have q(z) = 0 anyway, no?
- in the end, the objective is only a combination of ELBo + mutual information? Could you explain a little bit why is this novel? Especially, what is the difference with [1]?

[1] MIM: Mutual Information Machine (Micha Livne et al.)
limitations: nothing to report
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a unified solution to two problems which can occur when training VAEs - the hole problem (where the aggregated variational distribution fails to fit the prior) and posterior collapse (where the variational distribution becomes the same for every data point and is therefore uninformative).

The proposed solution is a modification to the VAE's training objective, where the per-data-point KL divergence term is replaced by the KL divergence from the aggregated posterior to the prior. This aggregate KL divergence term is expressed as the sum of KL divergences over the dimensions of the latent variable.

The authors compare their method to various baselines, all of which were designed to solve the posterior collapse issue. The proposed method appears to address the posterior collapse problem, while suffering the hole problem to a lesser extent than the baselines.
strengths_and_weaknesses: Strengths:
- This is a conceptually simple method which appears to be effective at solving both the hole problem and posterior collapse when training VAEs.
- The authors do a good job at explaining the hole problem and posterior collapse, as well as the apparent trade-off between the two.
    - Figure 1 is particularly clear for this.
- The empirical results appear to outperform the baselines, in terms of having all of the latent dimensions being both 'active' and 'consistent' (as defined in Appendix C), as well as the latent representations and observations having a high amount of mutual information.
    - The latent space visualisations appear to indicate that the proposed method clearly outperforms the baselines at solving both the hole problem and posterior collapse.
    - In addition, the proposed method appears to consistently outperform baselines at generating interpolations between sentences.
- Although this may not be the most significant advance in generative modelling by itself, it does seem to be a piece of work that the community could easily build on in order to make incremental advances in the field.


Weaknesses:
- Although the authors explain the hole problem and posterior collapse well, the explanation of their actual method difficult is to follow. Some examples include:
    - Throughout, the use of $\mathbf{x}$ and $\mathbf{y}$ to refer to instances of the same variable is confusing. Why not just use $\mathbf{x}$ throughout as is commonly done in the literature, avoiding the introduction of unnecessary notation?
    - Although it is clear why we would expect the proposed method to solve the hole problem, it is not explained why we would expect it to solve posterior collapse? Surely, as with the vanilla VAE, there is a local optimum where $q_{\phi}(\mathbf{z}|\mathbf{x}) = p(\mathbf{z}) \forall \mathbf{x}$? 
    - When reading the paper from beginning to end in order, it is unclear what the point is of Equation (3) and the paragraph just preceding it. I believe the clarity would be improved were this part moved to the part before Equation (8).
    - In Figure 1, should the $p_{\phi}$ terms not be $q_{\phi}$?
- The evaluation in the experiments section is not entirely convincing.
    - There are several missing details, e.g.
        - Are the results in Table 2 computed on the test set?
        - How many samples are used for the evaluation?
        - Why not report the (importance weighted) ELBO taken with a large number of samples, as is commonly done in the literature?
    - The interpolation task used to measure the ability of the model to do latent-guided generation isn't totally convincing.
        - The examples shown in Figures 6 and 7 are extremely short sentences, and at least qualitatively it is not clear that the DG-VAE is better than the $\beta$-VAE.
        - Why only show examples of the $\beta$-VAE and not the other baselines?

UPDATE: Increased score post rebuttal.
questions: - Is your method the first which intends to jointly solve the hole problem and posterior collapse?
    - If so, this would be a valuable statement to add.

Other questions included in the strengths and weaknesses section.
limitations: - The authors do not discuss the limitations of their method.
    - Arguably, the most important limitation of the proposed method is that the training objective is no longer a lower bound on the true log likelihood of the data. Does this mean that the model is no longer suitable for tasks such as density estimation, out-of-distribution detection, etc. which the vanilla VAE is otherwise useful for?
review_title: null

Meta-review: metareview: The paper addresses the KL collapse of VAE models by proposing a new regularization. Reviewers generally acknowledge the novelty of the work and have the tendency of recommending acceptance.
confidence: Certain
Decision: Accept