Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper  investigates a problematic behavior of current VAE models that lies in the inconsistency of current amortized encoder networks in the sense that the latent representations of semantically similar transformations (translation, rotation etc) of image data are dissimilar. Moreover, it proposes a tractable variant of the ELBO objective that is based on data augmentation and adds a KL-regularization term that penalizes the discrepancy in the posterior of the original and the transformed image data. The VAEs trained with this objective exhibit significant benefits (in terms of the utilization of the latent space and  marginal likelihood ) when compared to naive data augmentation demonstrating sota results in image generation and classification tasks.
main_review: The paper is very well written and easy to follow. The new ELBO variant is intuitively justified and applicable to all exisiting VAEs. I really enjoyed reading this paper. I only have a few questions on the experiments that I feel they need to be clarified and some suggestions for further applicability of the consistency-driven ELBO variant.

(1) Could the authors make the computational demands, especially on NVAE more clear? How many GPUSs are being used? It is stated that training takes 1 day approximately for all the experiments. However, this is inconsistent with the training time reported in the NVAE paper: (Table 6 in the appendix https://arxiv.org/abs/2007.03898) the training takes 55h so, given the augmentation I would consider that the consistent vae requires ~ 100 h?? Also, how many GPU-s are you using?

(2) I also believe that especially for the deep VAEs which make use of very large latent spaces, including AU as you do in Table 1 as well as the KL divergence (how much higher is it compared to NVAE without regularization) would help clarify that the benefits come from the better use of the latent space.

(3) How many samples (S in Algorithm 1) are you using when approximating the expected value?

(4) I believe it would be really interesting (albeit not required for acceptance) if the authors demonstrate consistency on more sophisticated encoders, beyond Gaussians, such as those based on normalizing flows.

(5)In evaluation metrics, it is mentioned "we also evaluate the effect of the proposed generalization to unseen data". I think it would be very informative if the authors also demonstrate another aspect of inconsistency in VAEs-- how well it would perform on data generated by its own generative model. 
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper improves VAEs by ensuring that the learned encoder maps augmented inputs $\tilde{x}$ and unaugmented inputs $x$ to the same representations. To do so, they minimize the KL divergence between the encoder conditioned on $\tilde{x}$ and $x$. They empirically show that such consistency regularised VAE (CR-VAE) achieves very strong density modeling results.
main_review: **Overall**: This is a clear paper that provides a very simple and useful method to improve VAE’s. My main issue is that the authors motivate the work as a way to learn useful representations but they (1) do not appropriately discuss related work in representation learning; (2) do not provide empirical evidence that supports the usefulness of the CR-VAE's representations compared to standard baselines in representation learning. Overall, I still think that the proposed method will be useful and interesting to people that work with VAEs.

**Strengths**
- **Very simple method**: the proposed method is very simple, which means that it can easily be applied to any VAE in a few lines of code.
- **Very clear paper**: the paper is very clear and well written.
- **Strong results for density modeling**: the log-likelihood results on MNIST / CIFAR / Celeba are strong, and the gains are significant compared to only using data augmentations.  

**Weaknesses**
- **Missing related work**  this paper is essentially trying to learn representations that are invariant to standard augmentations. As a result, this makes the proposed method very similar to self-supervised learning (SSL). For example, the same regulariser already appeared in [1].  This decreases somewhat the novelty of CR-VAE as these ideas have been well investigated for representation learning. From my perspective, this paper uses ideas from SSL to improve VAE’s. This is not inherently an issue as long as SSL is correctly discussed (note that lines 119-121 won’t be true anymore). Specifically:
    - [1]is very related to your method and uses the same regulariser. The key difference is that instead of reconstructing the image (learning p(x|z)) they use contrastive learning to maximize the mutual information I[Z, X]. This exemplifies well the relation between your method and SSL.
    - Most SSL methods aim to make representations of an augmented and unaugmented example very similar. For example [2] does that by minimizing the variance instead of KL divergence (see their Appendix A for relation to KL divergence).  Similar ideas of learning approximately invariant representations go way back, e.g., [3] from 2006.
    - [4] seems to be another very related work (although online since the beginning of March so contemporary). In particular, the Variational Invariant Compressor (sec. 4.1.) is essentially a VAE that takes as input augmented inputs but reconstructs unaugmented inputs to ensure “consistency” of the encoder. 
- **Lack of results showing the usefulness of the learned representations**: an important motivation of the work seems to be about learning useful representations, yet empirical results do not really support this claim:
    - the experimental results of downstream linear classifications are extremely weak compared to typical SSL methods, which suggests that the learned representations of CR-VAE are not useful for downstream tasks. Indeed, the best CIFAR-10 accuracy in this paper is 71.4% while standard SSL with linear evaluation achieves more than 94% (e.g. SimCLR Appendix B.9.).
    - The authors evaluate representation learning using mutual information, but (1) they do not compare to typical representation learning baselines; (2) it is well known that higher mutual information does not mean better representations [5].
- **Somewhat incomplete experimental results**: Given that this is a completely empirical paper (based on sensible intuition) I find the experimental section to be somewhat incomplete. Here are results that I would have liked to see:
    - (As previously said) a result that shows the usefulness of representations
    - Lineplot showing for one dataset (e.g. CIFAR10) the effect of $\lambda$ on the log-likelihood of CR-VAE, that would help to understand the robustness of the model to the additional hyperparameter.
    - Experiment analyzing the impact of the choice of augmentations on CR-VAE, to understand the importance of the choice of augmentations.
    - Standard errors for table 2/4/5/6. Especially table 5 as you claim SOTA.
    - Qualitative samples from a CR-VAE for 2D images to understand the effect of the regulariser.

**Suggestions / specific issues**
- I’m not fond of your use of “consistency”, which means something very specific in (frequentist) statistics and the title could thus be understood as a method to ensure that the variational posterior converges to the true posterior (similar to [6]). I would at least consider using another word. I understand if the authors ultimately keep it given that it’s (unfortunately) the term used in semi-supervised learning.
- The paper [7] is in the references but never actually cited in the text. Given that this paper shows comparable results on CIFAR10 and really shows the importance of data augmentation for density modeling, I think that it should be correctly discussed in your related work section.
- Line 202 please explain in the text why more active units should be better

**Questions**
- Why did you choose the KL divergence in this direction? Please explain in the text, and ideally add an experimental comparison.
- Another way of forcing “consistency” of the representations would be to augment the input but reconstruct the unaugmented examples (as in [4]), Have you considered this instead of your regulariser? I’d be curious to hear what you think, although I wouldn’t be surprised if “unaugmenting” the inputs is harder to do in practice.

**Minor Suggestions**
- Line 18: state of the art performance -> state of the art log-likelihoods          (I initially thought that SOTA performance was on downstream classification)
- figure 1: I would add the sub captions for those that skim the paper: (a) VAE   (b) VAE with augmentations   (c) CR-VAE (ours)
- Line 71 / 91 : e.g. -> e.g.,
- Line 82: is unable -> may be unable
- Line 148: will conduct -> conduct 
- Line 148 will apply -> apply
- Line 185: Finally -> Finally,
- Line 191: likelihood -> likelihoods
- Equation 5 : KLs use different fonts
- Line 198: metrics -> metric
- Line 199/200: using different quotes on left and right of “active”

**References**
- [1] Federici, Marco, et al. "Learning robust representations via multi-view information bottleneck." arXiv preprint arXiv:2002.07017 (2020).
- [2] Zbontar, Jure, et al. "Barlow twins: self-supervised learning via redundancy reduction." arXiv preprint arXiv:2103.03230 (2021).
- [3] Hadsell, Raia, Sumit Chopra, and Yann LeCun. "Dimensionality reduction by learning an invariant mapping." 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). Vol. 2. IEEE, 2006.
- [4] Dubois, Yann, et al. "Lossy Compression for Lossless Prediction." arXiv preprint arXiv:2106.10800 (2021).
- [5] Tschannen, Michael, et al. "On mutual information maximization for representation learning." arXiv preprint arXiv:1907.13625 (2019).
- [6] Wang, Yixin, and David M. Blei. "Frequentist consistency of variational Bayes." Journal of the American Statistical Association 114.527 (2019): 1147-1161.
- [7] Jun, Heewoo, et al. "Distribution augmentation for generative modeling." International Conference on Machine Learning. PMLR, 2020.
limitations_and_societal_impact: As previously stated, the authors need to discuss the limitation of CR-VAE's representation compared to standard representation learning methods. 

Potential societal impact is adequately discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a novel method of using data augmentation when training Variational Autoencoders. Given a family of semantics-preserving transformations, the idea is to enforce the invariance of latent code w.r.t. these transformations by adding a KL penalty term. This penalty improves the quality of learned latent representation and of the generative model significantly.


main_review: The paper proposes a novel method of using data augmentation when training Variational Autoencoders. Given a family of semantics-preserving transformations, the idea is to enforce the invariance of latent code w.r.t. these transformations by adding a penalty term. The penalty is KL divergence between encoder distribution at the augmented data point and encoder distribution at the original point.

Authors do extensive tests of the impact of this penalty on the quality of learned latent representations. Most experiments are performed on image datasets with augmentations being shifts, rotations and scaling. A variety of VAE architectures show improved quality of latents, compared to the standard way of applying augmentations. Quality of representations is accessed using mutual infromation estimates, active latent units and linear classifier accuracy (trained on latent features)
Authors also find that proposed method leads to better generative models with higher log-likelihood.

The proposed idea is original in the VAE context. The paper is well-written and experiments convincingly demonstrate the value of the proposed penalty.

Due to it's simplicity and effectiveness the proposed method will likely be widely used in practice. 

*****************************
Post-rebuttal

I'd like to thank the authors for their response and will keep my already high score the same.
limitations_and_societal_impact: The paper could be further improved by considering different families of augmentations and analysing the impact of the augmentation choice on latent representation and log-likelihood.

Another interesting extension to the proposed method could be to assume equivariance of latent code rather then invariance, by parameterizing learnable(e.g. affine) transforms acting on z variable and enforcing the equivalence of  x -> z -> z' and x -> x' -> z'.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a study on enforcing semantic consistency on VAEs. The authors show that the current VAEs are sensitive to semantical-preserving transformations and it will lower the quality of representations and generalization. To solve this issue, the authors propose a new algorithm, which first transforms a training sample into multiple semantically similar samples and then minimize the KL-divergence between the encoded distributions of original and new samples. Experiments on both 2D and 3D datasets show that the new algorithm is effective in increasing representation quality of VAEs.
main_review: Pros:
1. This paper discusses a very interesting problem, how to make representations semantically invariant.
2. The method is simple and can be applied to all variants of VAEs.
3. Experiments show the proposed algorithm is useful in a wide range of datasets.

Cons:
1. This paper looks like a direct application of contrastive learning on VAEs. Though the loss function is new,  the idea, algorithm, and even transformations are very similar to works on contrastive learning. The authors mention CAEs in line 130. However, more advanced contrastive methods are not compared as baselines.
2. Some experiment results are not significant. 
  (a) In Table2, the accuracy increase of using CR on MNIST is marginal. 
  (b) In Table 3, the differences of estimated log-likelihoods are also small in most settings.
3. The choice of some parameters is not verified. 
  (a) The most important parameter \lambda. I cannot figure out how to choose it. The authors say that ablation studies are performed, but I still cannot find the results.
  (b) The transformations (including translation, rotation, and scaling) and their parameters fully depend on handcrafting. Though it is a common problem for contrative learning, the authors should think about how to do it more automatically.

Other comments:
1. A large portion of this paper is devoted to introducing variational autoencoders and other background knowledge. For example, the first several paragraphs in the introduction and method part. The authors should talk more about their own staff with a very brief introduction of necessary backgrounds.
limitations_and_societal_impact: The authors mention that this paper might be used to generate artificial data or introduce biases. However, I think these are common problems for all deep generative models. So I don't think the authors need to improve this specific paper to solve the potential negative impacts.
review_title: null

Meta-review: This paper proposes an approach to improving the representations of generalization of VAEs using semantic-preserving transformations and a regularizer that enforces a congruence between representations of the original and transformed inputs.  This is shown to provide notable performance improvements for a variety of VAE models, with these gains, critically, larger than those achieved using simple data augmentation.

Overall, the reviewers were quite positive about the work, praising its simplicity, clarity, and empirical results.  Some issues were raised about the fact that most aspects of the paper are relatively straightforward applications of previous techniques, with the novelty predominantly originating only from their use in a VAE context, and insufficient discussion and comparison to other representation learning approaches that utilize semantic-preserving transformations, such as contrastive learning approaches.  However, all reviewers felt that the paper still deserved to be accepted in spite of these (albeit in some cases only marginally).

I believe that this assessment by the reviewers is quite fair and that, though not the most spectacular of papers, it is good solid work that will be of interest to the community and likely see some practical usage.  I thus recommend accepting it to the conference.

That said, I do think it is important that the authors make sure to make updates and improvements to the work, as per the reviewer's suggestions, for the camera-ready version.  In particular, I think the suggested additional discussion of related work is very important to add.  I would also strongly encourage the authors to include some non-VAE self-supervised baselines: though I mostly buy the argument that VAEs are important in their own right and do not think beating these baselines should be a condition for the work to be acceptable, I think it is still important to quantify how the approach compares to natural alternatives practitioners might consider (e.g. to see if there are aspects that this VAE-based approach is relatively good and bad at and how far off they are to SOTA approaches in the area).  Specifically, given much of the utility of the work is empirical, I feel that it is important that it properly investigates how relatively well VAE based approaches can perform in the target setting where semantically equivalent inputs can be generated.
Decision: Accept (Poster)