Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper addresses the problem of unsupervised model selection for disentangled representation learning. Based on the understanding of “why VAEs disentangle” [Burgess et al. 2017, Locatello et al. 2018, Mathieu et al. 2019, Rolinek et al. 2019], the authors adopt the assumption that disentangled representations are all alike (up to permutation and sign inverse) while entangled representations are different, and propose UDR method and its variants. Experimental results clearly show that UDR is a good approach for hyperparameter/model selection.
Overall, I think a reliable metric for model selection/evaluation is needed for the VAE-based disentangled representation learning. According to comprehensive experimental studies performed in this paper, UDR seems to be a potentially good choice.

However, I am not sure if very good disentangled representations must benefit (general) subsequent tasks, though the authors provide experimental evidence on fairness classification and data efficiency tasks. Actually, the data generation process in the real-world may consist of different generative factors that are not independent of each other. Though good disentangled representation provides good interpretability, it needs not to be better than entangled representation for concrete tasks. Specifically, for concrete supervised classification tasks, VAE with beta smaller than 1 (not towards disentanglement) might be the best (Alexander A. Alemi et al. 2017, Deep VIB).

Another concern is about the choice of some key “hyperparameters”.
For the KL divergence threshold in equation 3, you set it to be 0.01. It looks like the choice would control how much the UDR favors a “sparse representation map”. The larger the value, the few “informative dimensions” would be considered.
In supplementary material, you say that “uninformative latents typically have KL<<0.01 while informative latents have KL >>0.01”. Is this judgment based on “qualitative feeling”? For me, as you are contributing a ``quantitative measurement”, it is interesting and important to see how this threshold would generally affect UDR’s behavior in one (or more) datasets you have tried. 
Another hyperparameter I cared is P (number of models for pairwise comparison). In the paper, you validate the effect of P in the range [5,45]. How would P smaller than 5 affect UDR? According to Table 1, if I was using UDR, I’d rather using P>=20 (or at least 10) rather than 5.
Also, it seems to me P would grow up due to the size of factors that generate the data. Thus, I also have a little concern about the computation cost of the proposed metric (as also mentioned by the authors).

Others concerns:
-- As a heavy experimental paper, most experimental results are in supplementary material, while the authors spent a lot of time in the main text explaining the conclusions found in other papers.
-- To validate the fundamental assumption of UDR, the authors might consider to quantitatively validate that, disentangled representations learned by those approaches you used in the paper are almost the same (up to permutation and sign inverse). 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper proposes a criterion called Unsupervised Disentanglement Ranking (UDR) score. The score is computed based on the assumption that good disentangled representations are alike, while the representations can be entangled in multiple possible ways.  The UDR score can be used for unsupervised hyperparameter tuning and model selection for variational disentangled method.

The problem this paper focuses on is essential because we usually apply unsupervised disentangled methods to analyze the data when the labels are unavailable. However, existing metrics for hyperparameter tuning and model selection requires ground-truth labels. This paper allows measuring model performance without supervision, making the hyperparameter tuning and model selection possible in practice.

It looks like some parts of this paper need rewriting. In the abstract, it is not mentioned at all what is the proposed approach. Most paragraphs in the introduction section review the related work and background but do not introduce what assumption and strategy the proposed method adopted.

It looks like the proposed UDR is theoretically supported by Rolinek et al. (2019). However, the proof given by Rolinek et al. (2019) is for $\beta$-VAE, where the regularization can be turned into the constraint on KL divergence. I do not think the "polarised regime" holds for other disentangled model, for example, TCVAE, where a biased estimation of total correlation is introduced in the objective function. Therefore, I am not convinced that I should trust the results of the UDR, which combines multiple disentangled models.

The computational process of UDR is heuristic and somewhat arbitrary. There is no theoretical guarantee that UDR should be a useful disentanglement metric.  Although the UDR is supported by some experiments, I am not convinced that it is trustworthy for more complex real-world datasets.

Equation (3) looks problematic. Note that it is possible to train a Bidirectional Generative Adversarial Network (BiGAN) that can generate complex images based on a uniform distribution (Donahue et al., 2016). The encoder of the BiGAN can be considered as the inverse of the generator, which maps images back to the uniform distribution. This suggests that under the encoder-decoder framework, it is possible that latent variables can be informative even the posterior distribution matches the prior distribution. Although VAEs are trained using a different strategy, I do not see why the posterior needs to diverge from the prior distribution for informative latent representations. The encoder might simply be the inverse of the decoder under a certain scenario.

In summary, this paper focuses on solving an important problem. However, the proposed method is not well supported by theorems as it seems. The paper also appears to contain minor technical issues. Therefore, I am inclined to reject this paper.

References
Donahue, Jeff, Philipp Krähenbühl, and Trevor Darrell. "Adversarial feature learning." arXiv preprint arXiv:1605.09782 (2016).
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a metric for unsupervised model (and hyperparameter) selection for VAE-based models. The essential basis for the metric is to rank the models based on how much disentanglement they provide. This method relies on a key observation from this paper [A] viz., disentangled representations by any VAE-based model are likely to be similar (upto permutation and sign).

I am inclined to accept the paper for the following reasons:
1. The proposed approach is clear and easy enough to understand and well motivated
2. The paper has clearly outlined the assumptions and limitations of their work
3. The reported result show that models ranked by disentanglement correlate well with the supervised metrics for the various VAE models.
4. This metric is unsupervised and thus can utilize far more data than the supervised metric methods and can be useful even when the dataset has no labels.
5. The supplementary material also shows that the metric correlates well with the task performance.

[A] Variational Autoencoders Pursue PCA Directions (by Accident), CVPR 2019

---

Update:

Thanks for the thoughtful rebuttal by the authors to all the reviewers' feedback.

Based on the several discussions by the other reviewers and the discussion that happened, I am inclined to lower my scores to a weak accept. 
review_title: Official Blind Review #2

Meta-review: The authors address the important and understudied problem of tuning of unsupervised models, in particular variational models for learning disentangled representations.  They propose an unsupervised measure for model selection that correlates well with performance on multiple tasks.  After significant fruitful discussion with the reviewers and resulting revisions, many reviewer concerns have been addressed.  There are some remaining concerns that there may still be a gap in the theoretical basis for the application of the proposed measure to some models, that for different downstream tasks the best model selection criteria may vary, and that the method might be too cumbersome and not quite reliable enough for practitioners to use it broadly.  All of that being said, the reviewers (and I) agree that the approach is sufficiently interesting, and the empirical results sufficiently convincing, to make the paper a good contribution and hopefully motivation for additional methods addressing this problem.
Decision: Accept (Poster)