Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper is well constructed and written. It consists of a number of broad ideas regarding density estimation using transformations of autoregressive networks. Specifically, the authors examine models involving linear maps from past states (LAM) and recurrence relationships (RAM). 

The critical insight is that the hidden states in the LAM are not coupled allowing considerable flexibility between consecutive conditional distributions. This is at the expense of an increased number of parameters and a lack of information sharing. In contrast, the RAM transfers information between conditional densities via the coupled hidden states allowing for more constrained smooth transitions.

The authors then explored a variety of transformations designed to increase the expressiveness of LAM and RAM. The authors importantly note that one important restriction on the class of transformations is the ability to evaluate the Jacobian of the transformation efficiently. A composite of transformations coupled with the LAM/RAM networks provides a highly expressive model for modelling arbitrary joint densities but retaining interpretable conditional structure.

There is a rich variety of synthetic and real data studies which demonstrate that LAM and RAM consistently rank amongst the top models demonstrating potential utility for this class of models.

Whilst the paper provides no definitive solutions, this is not the point of the work which seeks to provide a description of a general class of potentially useful models.



review_title: Solid description of a general class of autoregressive density estimation models with potential utility

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose to combine nonlinear bijective transformations and flexible density models for density estimation. In terms of bijective change of variables transformations, they propose linear triangular transformations and recurrent transformations. They also propose to use as base transformation an autoregressive distribution with mixture of gaussians emissions.
Comparing with the Masked Autoregressive Flows (Papamakarios et al., 2017) paper, it seems that the true difference is using the linear autoregressive transformation (LAM) and recurrent autoregressive transformation (RAM), already present in the Inverse Autoregressive Flow (Kingma et al., 2016) paper they cite, instead of the masked feedforward architecture used Papamakarios et al. (2017).
Given that, the most important part of the paper would be to demonstrate how it performs compared to Masked Autoregressive Flows. A comparison with MAF/MADE is lacking in Table 1 and 2. Nonetheless, the comparison between models in flexible density models, change of variables transformations and combinations of both remain relevant.

Diederik P. Kingma, Tim Salimans, Rafal Józefowicz, Xi Chen, Ilya Sutskever, Max Welling: Improving Variational Autoencoders with Inverse Autoregressive Flow. NIPS 2016
George Papamakarios, Theo Pavlakou, Iain Murray: Masked Autoregressive Flow for Density Estimation. NIPS 2017

review_title: Comparison with MAF missing in several Tables

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper offers an extension to density estimation networks that makes them better able to learn dependencies between covariates of a distribution.

This work does not seem particularly original as applying transformations to input is done in most AR estimators.

Unfortunately, it's not clear if the work is better than the state-of-the-art. Most results in the paper are comparisons of toy conditional models. The paper does not compare to work for example from Papamakarios et al. on the same datasets. The one Table that lists other work showed LAM and RAM to be comparable. Many of the experiments are on synthetic results, and the paper would have benefited from concentrating on more real-world datasets.
review_title: Incremental work with unclear contribution

Meta-review: This paper looks at  building new density estimation methods and new methods for tranformations and autoregressive models. The request from reviewers for comparison improves the paper. These models have seen a wide range of applications and have been highly successful, needing the added benefits shown and their potential impact to be expanded further.
Decision: Reject