Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors applied residual learning machenism in VAE learning, which I have seen such methods in Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (https://arxiv.org/abs/1506.05751), which basically applied the residual learning method in GAN. But the authors fail to discuss the relationship and difference with this paper.
Also, the best paper from ICML 2019 claimed that unsupervised learning method can not really disentangle the features. They claim \beta-VAE, factor VAE is not good. The authors shall all discuess this point. Otherwise, it is not convincing to the readers.


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Overview:
Authors introduce a new VAE-based method for learning disentangled representations.
The main idea is to apply a “residual learning mechanism”, which resembles an autoregressive model, but here conditioning between sequential steps is done in both latent and input spaces. Namely, for each input, the method involves making several (one per *each* embedding dimension) sequential passes, where each pass takes the residual between the input and the current accumulated output (i.e. pixel-wise difference between two images) as well as the values of the sampled embeddings so far. 
Authors report quantitative and qualitative evaluation with a set of reasonable baselines (FactorVAE, beta-VAE), and the results seem to be slightly better.

Decision:
The writing quality is not great: there are frequent typos and not-so-precise formulations, and is at times hard to follow (see list below).
The method itself is not really well-motivated: there seems to be no formal justification provided, and the informal one is not very clearly explained in the paper. 
Scalability of the method is clearly an issue. Many applications might require the size of embeddings to have hundreds of dimensions, which would mean that the given method cannot really be applied for non-toy problems. 
With all those considerations in mind, I cannot recommend to accept this paper as is, thus the final rating, “reject”.

Additional comments / typos:
* It would be worth seeing how this approach relates to auto-regressive models.
* p2. “encoder maps … x to a latent representation q” - this sentence is not very strict. 
* p2. “.. is defined as following”: as follows?
* p2. “.. in addition of b”
* p3. “.. this reguralizer encourage”
* p7: “… it reduces the solution space and improve training stability“: is there any sort of theoretical reasoning that could support this?
* The reference format is not very readable, probably better to change to (Names, Year).

<update>
Authors addressed some of my concerns (thanks!), thus I increased my rating slightly.
There is still a large concern wrt to the computational complexity. I kind of understand the argument about the hyperparameter tuning, but it seems not really fair to compare to a bruteforce parameter search (one can probably do some sort of bayesian optimization).
</update>



review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The authors of this paper present a novel that for unsupervised disentangled representation learning. The model, named sequential residual VAE (SR-VAE), gradually activates individual latent variables to reconstruct residuals. Quantitative and qualitative experiments show that the proposed model outperforms beta-VAE and Factor-VAE.  Since the training involves a sequence of model training,  SR-VAE certainly consumes more time than other VAEs. Minors: citations in the main text should be put in brackets. 
review_title: Official Blind Review #2

Meta-review: This paper that defines a “Residual learning” mechanism as the training regime for variational autoencoder. The method gradually activates individual latent variables to reconstruct residuals.

There are two main concerns from the reviewers. First, residual learning is a common trick now, hence authors should provide insights on why residual learning works for VAE. The other problem is computational complexity. Currently, reviews argue that it seems not really fair to compare to a bruteforce parameter search. The authors’ rebuttal partially addresses these problems but meet the standard of the reviewers.

Based on the reviewers’ comments, I choose to reject the paper.

Decision: Reject