Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper mainly proposed the nonparametric variational autoencoder (NVAE) by 'combining' VAEs and the transformer.
Further, the transformer encoder and decoder are used in the VAEs framework.
Different from VAEs whose prior is the standard Gaussian distribution, the proposed NVAE applied the nonparametric variational information bottleneck (NVIB) regulariser for the latent embedding.
Finally, the experiment shows NVAE could do reconstruction, generation, and regularisation tasks. The interpolation further explored the meaningfulness of the learned representation.
strength_and_weaknesses: My biggest concern is in the experiment part. 

First, maybe I am wrong, but in my opinion, this work may be compared with SOTA works to support their claim. I am not convinced at least at the current reviewing stage. I am not convinced why we should choose NVAE rather than the original VAE. Because, the VAE could also do the reconstruction, generation, interpolation, etc. It is clear to me the VAE is more complex compared with VAE, with more inductive bias, but it is not clear to me NVAE whether NVAE could reach a significant improvement in performance or could do something VAE couldn't.

Second, the prior is a mixture distribution, could different distributions learn different aspects of the input object? e.g. a disentangled latent representation? Some ablation studies or visualizations may be needed. 
clarity,_quality,_novelty_and_reproducibility: This paper is very detailed. The author included dozens of experiment setups and math in the supplement material, which makes their claims convincing and reproducible.

The motivation for this paper is not clear to me. For example, in the first paragraph of the Introduction, I am not clear why we should combine the transformer and VAE. If the motivation is to combine the strength of transformers and VAEs, which aspect of VAE could be improved in theory? 
summary_of_the_review: As above, my main concern is about the experiment, and the second is the motivation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a probabilistic mixture model representation of the attention mechanism in transformers. The probabilistic representation  allows the authors to define priors and posteriors over the mixture distributions and ultimately redefine the transformer as a nonparametric variational autoencoder. They show experimental results for the reconstruction of input sentences, generation by sampling from the prior, regularize the size of the latent space, and interpolate.
strength_and_weaknesses: Strengths:
- The paper seems interesting and a rather novel concept. Their approach is sound and the results seem ok. There are no glaring flaws in the paper. 

Weaknesses:
- The point / takeaways / motivations are not very clear. Is the main benefit the regularization ability and the computational savings of having a lower dimensional latent space?
- While transfomer based models e.g. BERT / GPT-2 have been shown to be very useful, the paper is showing results on a a two layer Transformer encoder and decoder with a single attention-head. While this is the original proposed structure for using attention, it does not mimic either the encoder stack models e.g. BERT or the decoder stack models e.g. GPT-2 that are now the SOTA.
- Furthermore, it is not clear whether learning a NVAE for each attention mechanism (BERT would have 12*12=144 head) is computationally feasible and whether this training process is significantly more expensive than the standard learning of the query, key, and value weights. 
- In the experimental results, the NVAE model is compared against VTP and VT (which don't seem to have any hyperparameters as only 1 best model was chosen?) and a hand-coded solution VTS. The performance of NVAE and VTS seems relatively comparable, but is VTS supposed to be an "oracle" or is there any further description on what "hand coded" means?
- In tables 7 and 8, all generated samples seem equally senseless? Similarly the interpolation examples (tables 9-12) seem to show all models are comparable? 

clarity,_quality,_novelty_and_reproducibility: Clarity:
The grammar is kind of strangely worded, but overall fine. There is a lot of content, but sometimes key details seems to be glossed over (see above box).
summary_of_the_review: The paper is promising although it could be reworked to be stronger with a more clear focus and motivation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this manuscript authors embedding space of Transformer encoders as mixture distributions and as such are able to formulate transformer encoder - decoder as a VAE model. Number of mixture distributions is variable to authors decide to use Bayesian nonparametrics to solve the modeling issue, namely they use bounded Dirichlet prior. 
strength_and_weaknesses: Strengths: 
- Novel probabilistic formulation of attention model in transformers. 
- Full derivation is shown. 
- Model has potential usefulness beyond the experiments shown in the paper. 

Weaknesses: 
- Implicit reparametrization, that provides exact reparametrization for Dirichlet is not used ( https://arxiv.org/abs/1805.08498) 
- Manuscript seems to be typeset in a hurry, for example p(F|x) and p(F) are both named as prior. AFAIK, prior should not have data-term conditioning. Please clarify this point. 
- Even though mathematical development is convincing, authors have not motivated their work. They should clearly explain in Introduction that for what reason this development was done. It cannot be that just this formulation has never been done before!
clarity,_quality,_novelty_and_reproducibility: Paper needs more work in finalization. 
summary_of_the_review: All in all, I find the paper quite interesting and potentially very useful. I think the way how transformer is reformulated as a VAE model is quite neat. More large-scale experiments obviously would make the paper much better as authors also state in the Conclusions.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a VAE framework for that formulates attentions in transformers as a mixture of gaussians, where variational inference can be used to learn the framework. The proposed VAE with transformers as the encoder and decoder is used in text reconstruction and generation tasks. 
strength_and_weaknesses: Strength:

1. The (re) formulation of attention to a gaussian mixture and the application of variational inference are interesting and intuitive.

2. The paper provides comprehensive experiments on parameter sensitivity and ablation study.

Feedbacks (not all about weakness)

1. Although the formulation of a gaussian mixture of attention is interesting and contributes to the significance of the paper, I think the technical depth is not large enough for an accept yet. Using variational inference or VAE with gaussian mixtures is a well studied technique, which is directly applied given the formulation.

2. I don't see it is necessary to highlight Bayesian nonparametric or Dirichlet processes. First of all, DPs are usually applied into latent variables so that the dimensions of them can grow with data. In the case of this paper, the dimension of Z is the batch size of data, which does not usually grow in training or testing. Secondly, although discussing Bayesian nonparametric or Dirichlet processes, the paper ends up with a vanilla Dirichlet gaussian mixture model, which does not have strong connections with Dirichlet processes. It is not even a truncated approximation (e.g. stick breaking process) of DP. The paper names this unbounded DP and claims it as an approximation of DP, which requires theoretical justification.

3. In the experiments, the paper mainly compares with the baselines of different configurations of VAE with transformers. I think it is expected to put the proposed method in the category of text generation methods and compare it with other kinds of methods in the same category. This is to show how the method is positioned in the task.
clarity,_quality,_novelty_and_reproducibility: Clarity:

The general clarity of the paper is OK. But I do have confusions on connecting the ELBO with the attention architecture of the transformer. The ELBO shown in Section 3.1 is a general form of gaussian mixture. It is unclear to me that how to learn the parameters of attention by maximizing the ELBO. Clearer formulations of p(x|F) and q(F|x) with the attention parameters need to be introduced.
summary_of_the_review: The idea of the paper is interesting but I feel the technical depth is not enougth.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper has received borderline reviews (5-6-6-5). The reviewers expressed concerns about the experimental validation of the proposed extension and would have liked to see how it performs on larger transformer architectures. They would also have appreciated results against state-of-the-art models for text generation tasks. The authors argue that the experiments are rather a proof-of-concept that only demonstrates the validity of the theoretical developments proposed in the paper (i.e., the development of a non-parametric variational information bottleneck for attention-based representations, and the development of the proposed Transformer VAE using NVIB).

As an area chair, I believe the authors have properly argued their case and made an interesting contribution to both VAEs and Transformers. For these reasons, I tend to recommend accepting the paper.
justification_for_why_not_higher_score: Experiments remain a proof-of-concept.
justification_for_why_not_lower_score: Large-scale experiments would require extensively more computation. I believe the current paper already makes an interesting contribution to VAEs and Transformers. Other concerns of the reviewers have been addressed.
Decision: Accept: poster