Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**
This work proposes a new auto-regressive density estimator built using self-attention module from the popular Transformer network. TraDE can be seen as an extension of decoder-only Transformer network where an input embeddings are given by a simple RNN-based encoder. Like Transformer, TraDE leverages multiple layers of self-attention module to implicitly model long-range dependencies. This effectively eliminates the need for explicit vertex ordering and hence useful on data with no known canonical ordering. The proposed model is general and can be applied to both continuous as well as discrete data. Along with the MLE objective, TraDE is additionally regularised using MMD penalty which can be easily back-propagated using reparametrization / Gumbel softmax trick.

On standard benchmark dataset, TraDE produces better density estimates compared to recently established state of the art baselines. To further evaluate the qualitative performance of density estimators, it proposes suite of various tasks on which TraDE is shown to work well.

**Quality**
The paper is very well written and easy to follow. The experimental evaluation followed are standard (for density evaluation) and additional tasks depicts the usefulness of the sampled samples.

**Originality**
As summarized above, TraDE is a simple extension of decoder of Transformer. Minor modification like RNN-based inputs (inspired by Wang et. al) and MMD loss led to dramatic improvement for density estimation tasks. However, overall the work lacks novelty and I feel it is only simple integration of different building blocks that produces better density estimator.

Moreover, the tasks used for qualitative evaluation are also not completely novel. As pointed by the author, some of them (regression, two sample test, OOD) are already employed by the prior work.

**Significance**
Despite lack of novelty this work demonstrates,
1. better empirical results as well as qualitative evaluation using various real world tasks, and
2. carefully integrated self-attention module can perform better than many complex density estimators. 

**Clarity**
1. How do one sample data using TraDE. I do not find any mention of methodology employed for sampling.
2. Please include citations in Table 1.
3. From Table 3, I note that inclusion of MMD loss has very minor effect on performance. What is the training time tradeoff for including MMD loss ? Also compare quantitative results of (TraDE - MMD) model for various tasks. 
review_title: Transformer-based Density Estimator (TraDE)

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Summary
This paper uses the Transformer architecture for density estimation. It performs well on several non-trivial synthetic datasets and standard benchmark datasets. The authors also tested the model on other tasks that rely on density estimation.

## Pros
1. Addresses the issue of variable ordering on learning auto-regressive models and long-range dependencies with a new model architecture
1. Attempts to develop new evaluations other than log-likelihoods on practical uses of learnt density models
1. Impressive empirical results
1. Excellent written quality and comprehensive review of related literature.

## Cons
1. There is not much novelty. Simply borrowing the Transformer architecture does not seem sufficient for an academic venue
1. The additional evaluation tasks are not new, and the particular instances implemented have issues to be addressed
1. No mentioning of computational costs for training compared with benchmark methods.
1. Signs of heavy hyper-parameter tuning.


# Recommendation
Reject due mainly to a lack of novelty and some inadequate evaluations. I may raise the score slightly if the latter is addressed. 

# Issues and questions:
1. Lemma 1: it's not clear why "variables with a cut edge" means. Do you mean "variables the removal of which disconnects a graph"?
1. How can the model take the entire sequence but only return conditional distributions $p(x_i|x_{1:i-1})$? I think I may be missing something here. 
1. The equation of $alpha_j=\dots$ on page 4 is missing a parenthesis ")"
1. If the model is modelling each conditional correctly, why would it not model the joint well? This is mentioned in a strange place right after Lemma 3 which says it can model any joint.
1. All synthetic densities have a colour shift. A colour bar would help.
1. Introducing the MMD component does not seem to improve the log-likelihood much, so seems it is not essential for log-likelihood results. Can the author simply mention that adding this improves the model on small datasets?

## Questions regarding additional three empirical performance measures:
1. Why not conduct regression on all datasets? 
1. Two-sample testing is not novel, and the results can be reported at the same place as log-likelihoods. There are also a few statistical testing paradigms which are more rigorous than simply reporting a classification acc (e.g. Liu et al 20).
1. OOD: how is the threshold swept? A better quantity to report would be the area-under-the-curve (AUC).
1. Robustness to noise: adding noise should definitely affect test likelihood. Why do the authors believe that a better model should be more robust to changes in the training dataset?
1. An overarching question is: are all the benefit of the proposed method a result of introducing the MMD regulariser?
 
review_title: Density estimator with transformer architecture

final_score: rating: 3: Clear rejection
review_content: This paper proposes TraDE, a transformer-based density estimator that is capable of learning a density of real-valued tabular data. Compared to previously proposed transformers, there are three main differences in TraDE model: 1) the output is modeled as a mixture of Gaussians, 2) maximum mean discrepancy (MMD) is added to the loss, and 3) Gated Recurrent Unit (GRU) is used to provide positional encoding. Tested on a suite of benchmark tasks, the proposed method shows promising results over baselines.

Strengths:

The paper is the first to apply transformers on continuous-valued tabular data. The motivation of using transformers for auto-regressive modeling of continuous data stated in Section 3 is persuasive.

In addition to reporting the test likelihood, the authors conduct supplementary experiments to validate the effectiveness of the proposed model. The experiments include training a regression model with generated samples, two-sample testing using a classifier, detecting out-of-distribution samples, and learning on noise-corrupted data. These analyses provide a rich view of how the proposed model behaves and confirms the effectiveness of the proposed approach.

Weakness:

I reckon the contribution of the paper as a density estimation method is marginal. Compared to the previously proposed transformers, [1,2], The key architectural difference is the use of a mixture of Gaussian as an output distribution instead of a discrete distribution generated from a softmax function. This change of output parametrization seems trivial and straightforward, compared to architectural improvements presented in [1, 2]. Another difference is the use of maximum mean discrepancy (MMD) as a regularizer (or an auxiliary objective function). However, as shown in Table 3, the gain from the use of MMD is not consistent.

The analyses conducted to evaluate generative models (contribution no. 2 on page 1) are valid, but the paper is not the first to perform such experiments and therefore it is not adequate to claim those experiments as a core contribution. Examining the predictive performance of a model trained on generated samples is used to evaluated generative adversarial networks [3, 4]. Also, measuring out-of-distribution detection performance is used widely in generative modeling literature [5, 6].

Remark 4 on page 6 is wrong. The test likelihood is an estimator of cross entropy between the model distribution and the data distribution and therefore is connected to Kullback-Leibler (KL) divergence between them. As long as KL divergence is a meaningful measure of discrepancy, test likelihood gives meaningful information. Also, the maximizer of the objective in (1) is indeed p(x), given infinite data and a correctly specified model.

There is no explanation of how the outlier classes in Pendigits, ForestCover, and Satimage-2 datasets are defined.

There is no description of what transformers are used as the baseline in Table 3. Also, what is the "standard Transformer" used in Table 4? How exactly do these transformers differ from TraDE?

Minor comments:

- It would be more appropriate to use Proposition instead of Lemma for Lemma 1 and Lemma 3.
- Currently, the numberings of lemmas and remarks are confusing. I suggest to number them separately.
- It would be nice to mention that the datasets used in the experiments are tabular data, just in case if a reader is not familiar to the datasets.

[1] Katharopoulos, Angelos, et al. "Transformers are rnns: Fast autoregressive transformers with linear attention." arXiv preprint arXiv:2006.16236 (2020).  
[2] Child, Rewon, et al. "Generating long sequences with sparse transformers." arXiv preprint arXiv:1904.10509 (2019).  
[3] Ye, Yuancheng, et al. "GAN Quality Index (GQI) By GAN-induced Classifier." (2018).  
[4] Borji, Ali. "Pros and cons of gan evaluation measures." Computer Vision and Image Understanding 179 (2019): 41-65.  
[5] Du, Yilun, and Igor Mordatch. "Implicit generation and modeling with energy based models." Advances in Neural Information Processing Systems. 2019.  
[6] Grathwohl, Will, et al. "Your classifier is secretly an energy based model and you should treat it like one." arXiv preprint arXiv:1912.03263 (2019).
review_title: Review

Meta-review: This work explores an auto-regressive density estimator based on transformer networks. The model is trained via MLE with an additional MMD regularization term. 
Various experiments are performed on small benchmarks and show good results on density estimation. It is great to see that such a simple model is indeed very effective for density estimation on various small benchmarks (such as 2D density estimation and MNIST). 

The ablation experiments are informative and justify some of the model choices (such as the use of RNN to encode "positions"). Experiments are nicely chosen and paint a broad picture of the behaviour of the studied model.

The paper and author responses, however, excessively exaggerate the extent to which these results are relevant to the bigger picture in comparison to existing literature (e.g. flows and existing auto-regressive models).

As it has been extensively discussed with the reviewers, the proposed model is a straightforward application of a transformer network to auto-regressive modelling, this is specially so in light of existing work on auto-regressive models with transformers [e.g 1, 6, 8], self-attention [e.g 2]. BERT [7] itself can be used for auto-regressive modelling almost out-of-the-box (with the appropriate choice of masks during training).

At various points in the paper and author responses, it refers to flow models as "complicated/expensive" counterparts. These arguments are unfounded: auto-regressive models are particular cases of flows [3], and there are no obstructions to using transformer networks inside flows (in fact they have been already used, to achieve permutation equivariance and long-range correlations [e.g. 4]). 
The paper leaves comparisons to spline-flows out, arguing they are "hard to implement". This is quite conspicuous, as not only spline-flows are straightforward to implement, they produce results entirely on-par with the presented model (as an example, look at Fig 2 from [9] in comparison to Fig 1 from this paper).
Finally, the paper also misses an important discussion about the computational complexity of the proposed method. Auto-regressive models are considerably slower to sample from in relation to other types of directed models. Even more so with transformer networks as conditioners. For instance, flows [3, 5] allow for substantially faster sampling of large-dimensional data relative to auto-regressive models (by exploiting parallel sampling).


Extra comments:

The paper says "... Self-attention also enables
permutation equivariance and naturally enables TraDE to be agnostic to the ordering of the features ... "
This is true only for a *single* conditional $p(x_i | \text{Transformer}(x_{0 \ldots (i-1)}))$, not for the *joint* density. It is actually not straightforward to build auto-regressive models that are permutation invariant or that incorporate other forms of domain knowledge in general.
As an example, see [4] for how transformers and spline-flows can be used to produce exact permutation-invariant densities.


[1] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D. and Sutskever, I., 2020, November. Generative pretraining from pixels. In International Conference on Machine Learning (pp. 1691-1703). PMLR.

[2] Parmar, N., Vaswani, A., Uszkoreit, J., Kaiser, Ł., Shazeer, N., Ku, A. and Tran, D., 2018. Image transformer. arXiv preprint arXiv:1802.05751.

[3] Papamakarios, G., Nalisnick, E., Rezende, D.J., Mohamed, S. and Lakshminarayanan, B., 2019. Normalizing flows for probabilistic modeling and inference. arXiv preprint arXiv:1912.02762.

[4] Wirnsberger, P., Ballard, A.J., Papamakarios, G., Abercrombie, S., Racanière, S., Pritzel, A., Rezende, D.J. and Blundell, C., 2020. Targeted free energy estimation via learned mappings. arXiv preprint arXiv:2002.04913.

[5] Huang, C.W., Krueger, D., Lacoste, A. and Courville, A., 2018. Neural autoregressive flows. arXiv preprint arXiv:1804.00779.

[6] Sun, C., Myers, A., Vondrick, C., Murphy, K. and Schmid, C., 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference on Computer Vision (pp. 7464-7473).

[7] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova; ACL 2019.

[8] Child, R., Gray, S., Radford, A. and Sutskever, I., 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509.

[9] Durkan, C., Bekasov, A., Murray, I. and Papamakarios, G., 2019. Neural spline flows. In Advances in Neural Information Processing Systems (pp. 7511-7522).
Decision: Reject