Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposed a method for training Variational Ladder Autoencoder (VLAE) using a progressive learning strategy. In comparison to the generative model using a progressive learning strategy, the proposed method focuses not only on the image generation but also on extracting and disentangling hierarchical representation.

Overall, I think the purpose of this paper should be written clearly. It is not clear whether the purpose is learning the disentangled representation or the hierarchical representation. In my opinion, I think the focus of the proposed method lies in the hierarchical representation through progressive learning, but the experiments are involved more with disentanglement. Furthermore, I believe the authors need to explain the relationship between hierarchical representation and disentangled representation. In particular, it is not clear why learning hierarchical representation is helpful for disentangled representations.

The qualitative experiments are not convincing since the proposed model looks worse in both the reconstruction and hierarchical disentanglement for MNIST dataset than the base model VLAE, as shown in Figure 5 in [1]. Regarding the metric used in the experiments, the authors mention that the proposed disentanglement metric MIG-sup is what they first developed for one-to-one property, but it seems that it was already proposed in [2]. In addition, the proposed metric requires ground truth for the generative factors, so its usage is limited and not practical.

I think this work is similar to [3] in that both learn disentangled representations by progressively increasing the capacity of the model. I think the authors need to discuss about this work.

Ablation studies should be presented to verify the individual effects of the progressive learning method and implementation strategies on performance, respectively.

In Figures 2 and 3, the performance gap in the reconstruction error of the proposed method is greater than the base model when beta changes from 20 to 30. Therefore, it is necessary to show if it is robust against the hyperparameter beta. 

There is no definition of v_k in Equation (12), so it is difficult to understand the proposed metric clearly.

In summary, I do not think the paper is ready for publication. 

[1] Learning Hierarchical Features from Generative Models, Zhao et al., ICML 2017
[2] A Framework for the Quantitative Evaluation of Disentangled Representations, Eastwood et al., ICLR 2018
[3] Understanding disentangling in beta-VAE, Burgess et al., NIPS 2017 Workshop on Learning Disentangled Representations


-------------------------------------
After rebuttal:

Thanks for the revision of the paper and the additional experiments.

The authors' comments and further experiments address most of my concerns. In particular, new experiments show that pro-VLAE performs quantitatively and qualitatively better than VLAE. Also, Figure 10 and the result of the information flow experiment using MNIST show that the first layer learns the intended representations properly.

I appreciate the authors’ efforts put into the rebuttal, and the results of additional experiments are reasonably good. Therefore, I increase my final score to 6: Weak Accept.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper introduce pro-VLAE, an extension to VAE that promotes disentangled representation learning in a hierarchical fashion.
Encoder and decoder are made of multiple layers and latent variables are not only present in the bottleneck but also between intermediate layers; in such a way, it is possible to encode information at different scales, hence the hierarchical representation. Latent variables can be learned in an incremental way, by making them visible to the whole model progressively, so that as more latent variables become available, they encode lesser and lesser abstract factors.

Experiments are carried out on two benchmarks for disentanglement with annotations and pro-VLAE is compared to other methods in the state of the art.
Here, the authors introduce an extension of the Mutual Information Gap (MIG) metric, namely MIG-sup: it penalizes when multiple generative factors are encoded in the same latent variable. Qualitative results are also shown for 2 non-annotated datasets.

PROS
- The idea is fresh, well explained and experiments are sufficiently thorough. The novelty introduced is enough, provided that not much literature has explored progressive representation learning in the context of disentanglement.
- Results suggest that this is a promising direction for disentangling representations as pointed out by the authors in the conclusions.
- We appreciated the smart solutions for what concerns the implementation and training stabilization.

COMMENTS/IMPROVEMENTS
To improve the quality of the paper, consider the following comments:

- For the sake of completeness, experiments on Information flow should be also quantitative: it would be interesting to see how the information is captured by the latent variables on average on multiple runs, possibly trying different numbers of latent variables z_i.
- In sec 3.1 "z from different abstraction" is too vague and should be better formalized.
- In sec 2: "the presented progressive learning strategy provides an entirely different approach to improve disentangling that is ORTHOGONAL to these existing methods and a possibility to augment them in the future.": you should change to 'different'.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper proposes an approach to incrementally learn hierarchical representations using a variational autoencoder (VAE). This is shown to be useful qualitatively and quantitatively in terms of disentanglement in the representations.

To learn the hierarchy, the authors use a ladder architecture based on variational ladder autoencoder (VLAE) but incrementally activate the lateral connections across the layers at varying depth of the encoder and the decoder. A vanilla VAE is first trained. Followed by adding stochastic later connections and then retraining the updated architecture. This combined with beta-VAE inspired upweighting of the KL term leads to learning a hierarchy of representations. Each level of the hierarchy, the representations are disentangled. 

Inspired by progressive GANs, the authors employ ````"fade-out" when traversing the hierarchy. 

The authors also introduce a new metric to capture the one-to-one mapping of the ground truth factors to the latent dimensions.

Ablation studies by varying/removing fadeout compared to incremental learning will be useful. Can fade-out (different weighting of each level) be added directly to VLAE without incremental learning? 

Overall the paper is well motivated and easy to read. The results look impressive and the learned hierarchy and latent traversals are convincing. A more thorough comparison with VLAE will make the paper stronger.


review_title: Official Blind Review #3

Meta-review: This paper proposes a novel way to learn hierarchical disentangled latent representations by building on the previously published Variational Ladder AutoEncoder (VLAE) work. The proposed extension involves learning disentangled representations in a progressive manner, from the most abstract to the more detailed. While at first the reviewers expressed some concerns about the paper, in terms of its main focus (whether it was the disentanglement or the hierarchical aspect of the learnt representation), connections to past work, and experimental results, these concerns were fully alleviated during the discussion period. All of the reviewers now agree that this is a valuable contribution to the field and should be accepted to ICLR. Hence, I am happy to recommend this paper for acceptance as an oral.
Decision: Accept (Spotlight)